I still get annoyed when critics write that this or that author has "broken through the limitations of the genre" or whatever. It's not that I'm one of these guys who insists detective stories or the space opera or the bodice ripper don't have limitations. It's more that the claims for the boundary breaking always seem to be exaggerated. In every case -- Elmore Leonard to Raymond Chandler to Stanislaw Lem to Iain Banks -- what you have is a very good example of the genre, not a breaking of its conventions or limitations.
But here's a more interesting question: Who are some writers who can do both -- write a best-seller AND write a literary novel. And I'm not talking about the oxymoronic "literary potboiler," by which reviewers mean "literate potboiler/thriller/etc." (Please don't say Graham Greene).
My candidate of the month is Kate Atkinson, who's written at least one decent literary novel, Behind the Scenes at the Museum, and two cracking detective stories (actually, the best I've read in a long time), in Case Histories and A Good Turn.