I vividly remember the night my father’s politics became clear as day. He was smirking coolly at a friend whose dander was up over their increasingly tense debate. As the friend seethed, my dad leaned in and said, “You still haven’t answered my question: What has the Democratic Party done for you in the last 25 years?” That’s when it hit me: I was a son of the ever-elusive black Republican.

It was the spring of 1993, and I was just a teenager. The family had been enjoying an evening with friends who’d gathered to socialize over good food, good music, and good conversation. I remember saying to my mother, “I didn’t know we were Republicans.” She quickly shot back, “We aren’t; your father is.” Good grief.

This was my introduction to the complicated case of black Republicans, who have become more visible in recent months. Last November, voters elected Tim Scott, Mia Love, and Will Hurd to Congress, each of them representing firsts for the party. Ben Carson, a conservative neurosurgeon who appears to be gearing up for a 2016 presidential campaign, has been a popular draw at GOP events since he criticized Obamacare in front of the president. Despite these national personalities and nearly half a million more blacks voting against President Obama in 2012 than did in 2008, the black Republican remains somewhat of an oddity.

This paradox is not just about the party’s platform. Republicans still struggle with the perception that their party is indifferent to the particular experience of blacks in America. There’s nothing exotic about black people who believe in free markets and small government, who oppose abortion, same-sex marriage, or affirmative action, whose views on immigration and foreign policy are imbued with a strong security-first mindset, or who ascribe to respectability politics and the concept of rugged individualism. They are black conservatives, and they are in ample supply. However, it is unusual to find black people who, in addition to holding these conservative principles, also oppose the Great Society-era statutes outlawing racial discrimination and deem them unnecessary, like many in the Republican Party. Taken together, black Republicans appear to espouse the party platform and accept the party’s view that structural racism does not play a role in denying black America access to today’s meritocratic society.

Black Republicans are outliers in the two camps to which they belong—the elephants in any room they enter. The friction between much of black America and the Republican Party, particularly regarding civil-rights legislation and social programs, makes the position of the black Republican all the more difficult. Recent polling provides evidence of the dissonance. For example, 66 percent of white Republicans believe blacks receive equal treatment in the criminal justice system compared to only 10 percent of blacks. On more controversial matters such as an apology for slavery and reparations, nearly 60% of black Americans support these measures whereas only 10 and 4 percent of Republicans do, respectively. Though racial polarization exists on these issues, the gap is widest between black Americans and Republicans. As a result, black Republicans find themselves as precarious conciliators between two groups that seem uninterested in reconciliation.

In short, black Republicans are perceived to be the token black person in a group of Republicans, and the token Republican in a group of black people. This sense of isolation has shaped the black Republican experience for decades. Their plight is chronicled exceptionally well in The Loneliness of the Black Republican, a new book by Harvard Kennedy School professor Leah Wright Rigueur. Her thorough examination traces the winding journey of black Republicans from the inception of the New Deal to the election of Ronald Reagan.

So why would any black person want to be Republican? It varies from person to person, but Rigueur’s work suggests the reasons fall into four rough categories. First, many inherited the political loyalty of their ancestors that supported the party of Abraham Lincoln and saw hope in the black outreach of Teddy Roosevelt. Second, others believe that economic security is the best way to secure civil rights, so their “work hard and be twice-as-good” mentality aligns well with conservative principles and policies. This may sound like prioritizing individualism over collective well-being, but it is closer to a belief that personal responsibility is the best way to achieve group progress. Third, there are those who believe a contested black electorate is the best way to empower black America, so they make the pragmatic decision to work from the Republican side. And fourth, and most interestingly, there are opportunists who choose the Republican Party because its black underrepresentation creates more attention and prospects for the few black members it does have.

Whatever the rationale, the end result is a political existence that can sometimes feel forlorn. Rigueur establishes that the black Republican’s loneliness is not solely a matter of policy differences with the black population or the party, but the byproduct of differing strategies on how to build a winning coalition of voters. She argues that black Republicans believed it necessary to walk the fine line between race-neutral and race-conscious presentations of the party’s conservative policies grounded in individualism, meritocracy, and hard work. In theory, this would allow the party to attract black voters without alienating the white blue-collar and middle-class constituents it desperately needed to win elections.

In practice, however, it was not so simple. Black voters generally support social programs and race-conscious civil-rights legislation to construct a colorblind, meritocratic society, but many white middle- and working-class voters tend to see these policies as unfair assistance. And since the white percentage of the electorate was far larger then than it is today, the party chose to build its strategy around the middle-class white voter rather than mount a serious challenge to the Democratic hold on the black vote. As a result of black Americans’ self-interest in civil-rights protections and the party’s cold electoral calculus, Rigueur writes, black Republicans “were simultaneously shunned by African American communities and subordinated by the Republican Party.”

The black Republican quandary, then, is the constant struggle to manage the “seemingly incongruous intersection of civil rights and American conservatism.”

The Republican Party’s investment in an electoral strategy that capitalized on racial divisions seemed to culminate in a move by 1980 presidential hopeful Ronald Reagan. As Rigueur notes, Reagan made the curious decision to deliver a speech on states’ rights in Philadelphia, Mississippi—most notable as the town of the Freedom Summer murders of 1964 where the Ku Klux Klan killed three civil-rights workers who were registering black voters. Reagan’s staging was both tone-deaf to black Americans and specifically attuned to the working-class whites in the Deep South. It is the nuance of actions like these that black Republicans like my father—a man steeped in the Protestant work ethic and respectability instilled by his minister-sharecropper father and the self-sufficiency mantra of the 1970’s black-power movement—have to explain to blacks and party leaders.

Today, the rollback of voting-rights provisions and affirmative action, growing challenges to fair-housing legislation, and new voter-identification laws—all in the name of creating a more fair society—effectively pit civil rights against conservatism. Just a few days ago, the Republican-led Senate Judiciary Committee decided to drop “civil rights” from the title of one of its subcommittees. Even Senator Rand Paul, the Republican most actively engaging black Americans today, has expressed concerns about the Civil Rights Act that harken back to 1964 Republican presidential candidate Barry Goldwater’s opposition to it. Black Republicans are left to straddle the schism once again.

Though black Republicans are not a monolithic group—Allen West is not Condoleezza Rice, for example—they largely have believed in the traditional conservative principle of small government that is exceptionally strong where it needs to be, such as in matters of national security. A key attribute of black Republicanism is the addition of civil-rights protections to the list. In this regard, most of them present as less conservative than their white compatriots, but do so in a party much too conservative on race issues to be palatable to most black Americans. This is the genesis of their loneliness.

As Rigueur and others have noted, President Obama’s politics were heavily influenced by the same cultural experiences that honed the beliefs of prominent black Republicans such as Senator Ed Brooke and General Colin Powell. Black politicians who want national appeal must harbor the principles often associated with conservatism: race-neutral frames, individualism, respectability, and hard work. This is certainly evinced in the campaign platforms of the current crop of black Republicans in, or seeking, national office. But whereas Obama was embraced by the Democratic Party and became its face, Powell was booed by delegates at the 1996 Republican National Convention and has been essentially ousted as the standard bearer for black Republicanism. A consensus heir has yet to appear.

If the Republican Party wants to attract black voters, the path starts with reconciling its past and recognizing the primacy of civil rights. A century of racially driven electoral strategies cannot be undone in one election cycle. From the state-level “lily-white” movement after Reconstruction to the post-civil rights era Southern strategy that realigned the two parties on race, the Republican Party made conscious decisions at the state and federal levels to court the white vote at the expense of the black vote. Reversing this process begins with guaranteeing black Americans that the party won’t undo hard-won federal civil-rights legislation. With the permanence of these laws assured, black Americans would be free to prioritize other policy issues. Conservatives could then find significant numbers of welcoming ears for their economic and domestic policy initiatives.

As for my Republican father, he is an Obama supporter who may be lonely in his party, but isn’t all that lonely in his politics. He, like a significant number of blacks, is simply a conservative who believes respectability is important, but wholly insufficient to provide black people a fair and just American experience. Black Republicans like him believe a strong, central government is required to ensure the protection of civil rights, particularly for those who have been historically denied access to them. For these black Americans, it is their partisan affiliation, not their politics, that isolates them from both their community and their party.