Her luck inevitably ran out. On April 10, 1864, her hard-won blue army uniform was now damning evidence; Walker was taken prisoner by Confederate soldiers under General Harvey Hill. Five days later, General Ulysses S. Grant ordered all women to leave Union battlefields, but by then Walker had been sent to Castle Thunder Prison. It was run by Captain George W. Alexander, a cruel man who regularly hanged prisoners by their thumbs, toes barely scraping the floor, primed for flogging.
The guards were equally infamous, particularly in the female ward where Walker would spend the next four months. She no doubt heard about another female spy had been chloroformed, raped, and killed there. Walker's reputation preceded her, and the imprisoned "female Yankee surgeon" who believed in the emancipation of African Americans was openly ridiculed in Virginia papers. "Miss Doctress, Miscegenation, Philosophical Walker, who has so long ensconced herself very quietly in Castle Thunder, has loomed into activity again," wrote a reporter in The Richmond Examiner.
In letters to her parents, Walker claimed she was well fed and treated fairly, but she was likely protecting them from the truth. Food was known to be scarce at Castle Thunder, and rations maggot-filled. Walker's mattress was infested. At night, there were screams, but the loudest sound came from below, as mice scurried across the filthy prison floor.
Both the Confederate and Union armies were desperate for physicians, and on August 12, 1863, Walker was exchanged, in equal measure, for a male physician. President Lincoln, eager to hear about her time as a prisoner-of-war, summoned her to Washington.
Walker had now served as a physician at Indiana Hospital, Bull Run, Warrenton, Fredericksburg, Chickamauga, Chattanooga, and Atlanta, but securing another commission required renewing the letter-writing campaign. She was finally sent to Louisville to be the head surgeon at the Female Military Prison, but it was populated by Confederate woman who rebuffed the "Yankee woman doctor," often refusing her services. By the time she managed to trade Louisville for Clarksville, President Lincoln had been assassinated, and the war was coming to a close. On June 15, 1865, Walker requested that her military service conclude. This time, the army readily granted her request.
The partial muscular atrophy Walker developed during the four months she spent as a prisoner-of-war never fully healed, which meant her career as a surgeon was over. She attempted to resume her medical practice at 374 Ninth Street in Washington, but found her patients had gone elsewhere in her absence. The Johnson administration, already under scrutiny for failure to address the post-emancipation hardships of African Americans, rejected her request to serve as a medical inspector in the Freedmen's Bureau. The presence of a woman doctor would have only intensified the criticism. Despite her service, Walker was unsuccessful in finding a postwar commission from the government, though formal recognition was in the works.