Ann Beattie‘s new book, Walks With Men, is her first novella, clocking in at just over 100 pages. She describes the appeal of the form in a brief interview with the York County (Maine) Coast Star:
Short stories distill language, which can be to the writer’s advantage; novels don’t exactly do the same thing in the same way, because it would get tedious. But in the middle range (the novella), the writer can sometimes trust in the reverberation of language and images and symbols in the same way the short story writer can trust in their inherent weightiness. In a novel — for me — language dissipates, in terms of subtly suggesting things, and other things have to take over.
In saying that, she references a 2003 Bomb interview of Steven Millhauser by Jim Shepherd, where he talks a little more colorfully about the appeal of the middle route:
The novella wants nothing to do with the immense, the encyclopedic, the all-conquering all-devouring prose epic, which strikes it as an army moving relentlessly across the land. Its desires are more intimate, more selective. And when it looks at the short story, to which it’s secretly akin, it says, with a certain cruelty, No, not for me this admirably exquisite, elegant, refined—perhaps overrefined?—delicately nuanced, perfect little world, whose perfection depends so much on artful exclusions. It says, Let me breathe! The attraction of the novella is that it lets the short story breathe.
Sounds nice, though pulling it off successfully is a tougher trick—Don DeLillo has been trying to do it for the past ten years, and only now, with Point Omega, does he seem to have a firm command of that middle length. Last month the Emerging Writers Network dedicated a month-long series of posts to the questions of what the form is and why/if it works. One of the best comments there comes from novelist Steve Stern: “So what if the novella denies you the primary intimacy with its characters that a novel affords; it enhances your awareness of the mystery of their movements, the allusiveness of their speech, while at the same time preserving your appreciation for the beautiful symmetry of the structure that contains them.”
markathitakis.com 
It’s interesting that the novella has such a problematic history within literary fiction, yet it is so vital to SF. The remaining SF digests and online magazines regularly publish novellas, and the Hugo, Nebula and World Fantasy awards for Best Novella garner a fair amount of attention. Even going back to the mid-century, it was a popular form among the Golden Age SF writers. The two volume Science Fiction Hall of Fame Vol. II, which is devoted to pre-1964 novellas, collects some of the best, including Jack Vance’s “The Moon Moth” – a personal favorite. I suspect one can attribute this dichotomy, at least in part, to the role that the pulps played in the development of SF.
Interesting point. My knowledge of science fiction is very surface-level, but I’m curious how much this was a case of form following function—meaning, that the format, economics, and page counts of the pulps made them more amenable to novella-length stories. Is it just as true with crime fiction, though? Seems like the big-name noir writers (Hammett, Goodis, Chandler) wrote either short stories or novels. “The Postman Always Rings Twice” is short enough to qualify as a novella, but it wasn’t originally published in a magazine.
I think that your analysis of “form following function” is spot on regarding why the novella played such a central role in the development of SF. It probably survived the end of the pulp era, in part, because SF fandom had embraced the form and continued to support forums that lent themselves to their publication – digests, glossies, fanzines and popular anthology series (like Ellison’s Dangerous Visions). Internet magazines like Clarkesworld have given the novella yet another wind in recent years.
I almost mentioned crime fiction in my original post. There are historical examples of crime novellas – you mention Cain, but I think that”A Study in Scarlet” is also novella length. And I can think of a handful of noteworthy crime novellas in recent years – Chabon’s The Final Solution, Black/Banville’s The Lemur. But, overall, the novella does not seem to have the same central role in crime fiction as it does in SF, despite the popularity of crime pulps back in the day.
Maybe the relative ages of the genres play a role? Here’s a theory: Crime pulps were popular, but the genre itself had been around awhile before they came about. SF, on the other hand, really developed in the pulps. While publishers were willing to invest in the crime novels of Christie, Hammett, etc., making the novel the defacto form for adult crime fiction, they did not have such faith in the upstart SF – a genre written for kids, often by kids. Thus, as the early SF writers started to experiment with longer forms, they had to do so within the confines of the pulps.
Novellas? Read Elkin.
Rick Moody’s “The Albertine Notes,” published in one of the McSweeney’s volumes edited by Michael Chabon, is an example of a fine novella that combines things sf does well with things postmodern literary fiction does well. It was arguably the best sf story published in 2002.