AT BAY by Ralph Mclnerny. Harper & Row, $8.95. Rogerson is a middle-aged professor who suffers from prostate trouble and an uncontrollable itch to throw monkey wrenches into other people’s works. The reader can hardly feel much sympathy when he drops one into his own. None of this business is as amusing as the author seems to intend.