Fights over literature are often just fights over genre. As proof, look to Christopher Beha's piece published last week in Slate. Beha is responding to Jennifer Weiner's campaign to get the New York Times Book Review to publish more reviews of popular books.
In the course of his discussion, Beha tries to make a distinction between genre fiction—sci-fi, mystery, romance, fantasy, etc.—and literary fiction. He concludes that literary fiction is in fact a genre itself, but there are other books out there, the books of real literary quality, that have no genre. He calls these "Holy Crap" books.
But if I’m willing to admit that there is such a genre as “literary fiction” and that books that fit easily into this genre aren’t inherently better than books that fit easily into any other genre, I hope that Weiner is willing to admit that an awful lot of fiction doesn’t fit easily into any genre, either because it is truly sui generis (which is rare) or because it combines various genres in new ways (which is less rare). When these books succeed they offer a level of richness and surprise that can’t be found in books that fit easily into genre classifications, including the genre classification “literary fiction.” Even when they fail, they tend to fail in interesting ways. But that’s not all: one characteristic of this sort of book is that it is often difficult at first glance to tell if it has succeeded or failed, because we don’t know what standard we ought to be using to judge it. A person who devours Sue Grafton’s alphabet novels may finish U Is for Undertow and think, “That was one of Grafton’s best,” or “That was sub-par Grafton,” but she almost certainly won’t finish it and think, “Holy crap, what was that about?”
For Beha, then, “genre” means adhering to certain "specifications and expectations"; genre is something inherent in books; it's about formal traits and qualities within the novel itself. Books that meet the specifications and expectations of science fiction are sci-fi, and those that meet the specifications and expectations of literary fiction are lit fic. Those that bust out of those specifications and expectations are something else: holy-crap books, or great books. They're different.
But this take on genre reflects a widespread misconception. Genres are not in fact necessarily based on expectations or specifications or elements inherent to the work. On the contrary, many folks who work on genre theory make a persuasive case that genre is much more amorphous. John Rieder, for example, argues that science fiction is not one thing or another, but a "web of resemblances" created by intertexual references. Jason Mittel goes even further, and argues that television genres are constituted basically by social and cultural agreement.
So genre is not just what's in the art, but where the art appears, what institutions support it, and all the other markers that cause people to decide that the genre is a genre. This is why efforts to define, say, comics on a formal basis so quickly devolve into nonsense, with respected authorities insisting that one-panel political cartoons don't count, or trying to define the form on the basis of continuing characters. They're looking in the wrong place, like trying to define an ocean ecosystem by looking solely at the plankton. (And yes, I know that comics are often thought of as a medium, not a genre—and that's a social convention too.)
That's why the argument that some books transcend genre is incoherent: Genres aren't conceptually solid enough to be transcended. Any genre is going to be made up of things that both fit and don't, and over time those things will change and shift. Frankenstein, as John Rieder argues, was Gothic romance first, but now it's science fiction. Jimmie Rodgers was hillbilly music, now he's country. Dave Sim, a well-known comic creator, wrote a story in prose, but sold it in comics shops, so it was treated as a comic. I first read about Philip K. Dick's Confessions of a Crap Artist in a magazine that reviewed sci-fi and fantasy books, because Philip K. Dick was a sci-fi writer, and so his book was (for these purposes) sci-fi. Such overlaps and blips aren't mistakes, nor are they necessarily signs of a work's unique brilliance. They're just a function of the fact that genre is a more like a jelly, or a vague description of a jelly, than it is like a box.