Akim Reinhardt argues as much as we would like to believe it, most of us wouldn't have been "the person working to free slaves through the underground railroad, or living peaceably with Indians, or smuggling Jews out of Europe." He explains:
You know who the abolitionists were? Not to paint with too broad of a brush here, but a lot of them were religious fanatics. They were the crazies, the radicals, the ones that everyone else pointed to and said: Hey, you’re really nuts. What the hell’s wrong with you? Knock it off already. Abolitionists were the ones who regular people mocked, jeered, and cursed. They were the outsiders of their day, the lunatic fringe of the early 19th century. Slavery was normal, so a society that largely accepted slavery labeled them as crazy.