Early next year, Ken Chenault will step down as the CEO of American Express, a position he has held for more than 16 years. His retirement won’t just change the uppermost ranks of the company he’s led—it will also whittle the number of black CEOs in Fortune 500 companies down to three, continuing a downward trend among the nation’s biggest corporations.

When Chenault took over at American Express in 2001, the number of CEOs of color at the country’s biggest companies was on the rise. Chenault (who was only the third black CEO to run a Fortune 500 company) was eventually joined by Don Thompson at McDonald’s, Ursula Burns at Xerox, Roger Ferguson at TIAA-CREF, Kenneth Frazier at Merck, and Marvin Ellison at J.C. Penney, among others. Between 2005 and 2011, Fortune 500 companies added eight new black CEOs. When their overall numbers peaked (at seven, in 2007), their representation was still minuscule, representing fewer than 2 percent of all Fortune 500 CEOs, but at the very least, during the 2000s, the direction of the trendline was up.

It wasn’t just black executives who were being promoted at this time. As the sociologists Richard L. Zweigenhaft and Bill Domhoff catalogue in The New CEOs: Women, African American, Latino, and Asian American Leaders of Fortune 500 Companies, other minority groups came to be better represented as well. During that same period from 2005 to 2011, 12 Latinos and 13 people of Asian descent were promoted to CEOs in Fortune 500 companies. There were 13 Latino CEOs in 2008, and then again in 2015; the number of CEOs of Asian descent peaked in 2011, at 14.

But that trend toward diversity later stalled out, then reversed. In recent years, the pace at which big companies appointed CEOs of color has slowed dramatically. As the CEOs who broke racial glass ceilings retired or were removed, their ranks were replenished primarily by white men, returning the country’s corporate leadership to an earlier status quo. It shouldn’t be incumbent upon a single company with a CEO of color to replace him or her with another person of color, but losing CEOs of color diminishes executive diversity when, among big corporations, they aren’t being appointed elsewhere.

It’s hard to point to a single cause of the reversal of ‘00s trends, but there are several theories. Zweigenhaft says that a crucial turning point was the financial crisis, which prompted layoffs even at high levels of companies. “After 2008 or 2009, often the people who were not white men were the first to go,” he says. “Women and people of color were more likely to get pushed out.” There’s data that broadly supports that idea, suggesting that a “last in, first out” model can apply to both minorities and women during times of economic hardship. A dynamic like this would produce even more of a shortage of people of color in companies’ senior ranks, further reducing the diversity of those next in line for the role of CEO.

Another possibility is that two broad pushes for diversity—one shorter-term, the other longer-term—began to falter at around the same time. In the shorter term, Zweigenhaft points to the limits of initiatives led by colleges and employers that made diversity a buzzword in the past decade: A large concerted attempt to integrate an industry can change the landscape just enough to make progress look substantial and successful—which then makes diversity a less pressing priority. A few years later, after some inattention, the issue resurfaces.

And then there is a theory that in the longer term, the wave of CEOs of color was the slow manifestation of the effects of the civil-rights movement—which allowed for some measure of educational and social equity, and put more people of color into higher-ranking positions and major companies. But without continued pressure of a similar scale, the theory goes, such progress waned.

Now, the future of CEO diversity is not very encouraging. In a 2015 analysis, Zweigenhaft referred to the current pace of diversification in the highest ranks of corporations as “glacial,” noting that the addition of one or two people of color to a few high-ranking positions at big companies only moves overall representation by a percentage point, if that. And the limited progress that has taken place in recent years has aided white women and South Asian immigrants much more than to blacks, Latinos, or Asian Americans from places such as China, Japan, or Korea, Zweigenhaft says. All the while, women of color remain virtually nonexistent in the highest ranks of large businesses.

While diversity pushes have led to the hiring of more young people of color at large companies, very few can be found in the senior ranks from which CEOs are often selected. In forthcoming research, Zweigenhaft and Domhoff studied the pipeline of potential future CEOs, defined as those executives who were one to three rungs below CEOs, and found that prospects for candidates of color were slim: More than two-thirds of potential future CEOs were white men, followed by white women, who accounted for around 20 percent. Asian men accounted for just under 8 percent of this group, and black and Hispanic candidates of both genders combined to make up less than 7 percent of the pool.

Even with more people of color attending prestigious schools and with a bevy of programs aimed at getting more minorities into management at the country’s biggest companies, advancement and retention within those companies remain elusive. It may have something to do with employees of color being likelier to leave a company before they reach its upper echelons, perhaps because of corporate culture or a lack of clear advancement opportunities. And some of the issue is likely attributable to the way senior executives select their protégés (and thus often their successors). Zweigenhaft says that in the most elite professional circles, mentees are often chosen based on how comfortable an existing team of executives or board members might feel with them. “If the leadership team is white men, they tend to look at people like themselves,” he says. And even at its most diverse, the leadership of major companies is overwhelmingly white and male.

What makes the outlook for improvement even more dismal is that the share of people of color at the highest levels of corporate America is decreasing even as the need for companies to cultivate diversity becomes more apparent. Without fundamental changes to companies’ hiring, retention, and mentoring processes, it’s hard to imagine that big business will ever get more diverse.

When Chenault steps down from American Express in 2018, he’ll be replaced by Stephen Squeri, who is currently the company’s vice chairman, and American corporate leadership will take one more step toward looking a bit more like it did before Chenault held the role.