Something is rotten in the U.S. economy. Poor men without a college degree are disappearing from the labor force. The share of prime-age men (ages 25-54) who are neither working nor looking for work has doubled since the 1970s.

The U.S.’s labor participation rate for this group of men is lower than every country in the OECD except for Israel (an outlier, because of the high number of non-working Orthodox Jewish men) and Italy (an economic omnishambles). Today, one in six prime-age men in America are either unemployed or out of the workforce altogether—about 10 million men.

So, this is the 10-million-man question: Where did all these guys go?

According to a report from White House economists released last week, non-working prime-age men skew young, are less likely to be parents, are disproportionately black and less educated, and are concentrated in the South.

In the last few years, several writers and economists have suggested that many of them are in school, on disability, or in prison. More optimistically, some said that men are more likely to help their spouses with raising children and cleaning the house. But upon investigation, none of these answers fully explains the disappearance of prime-age men.

1) Are they in school?

This would be one of the most benign—or even hopeful—reasons for the drop in male participation. Alas, it doesn’t seem to explain much.

Since 1990, the number of men over the age of 25 enrolled in a post-secondary institution has increased slightly, from 2.5 million to about 3 million, most of which was an increase among people enrolled part-time. Overall, a jump of 500,000 accounts for just a fraction of the growth of non-working men. (If the male participation rate hadn’t changed since 1990, there would be about 3 million more men in the labor force.)

The men most likely to drop out of the labor force today are those who never started college. This is a remarkable shift. Fifty years ago, college graduates and high-school dropouts were similarly likely to work, as one can see in the graph below. Today a high-school graduate who has never gone to college is four times more likely to drop out of the labor force than he was in 1964.

Men between 25 and 54 are much better educated than they were in 1964. That fact alone should have predicted a rising participation rate, since college graduates are more likely to work. Instead, the least educated men are abandoning the work force more than ever. That is the real mystery.

2) Are they on disability?

This is another common explanation for the drop in male participation. But again it doesn’t explain more than a fraction of the phenomenon.

There’s not much doubt that Social Security Disability Insurance takes people out of the workforce, often by inelegant design. In order to qualify for disability payments, people typically have to prove that they cannot work full-time. SSDI critics say this policy sidelines many people who might otherwise be able to contribute to the economy.

But how many people does SSDI really remove? From 1967 to 2014, the share of prime-age men getting disability insurance rose from 1 percent to 3 percent. There is little chance that this increase is entirely the result of several million fraudulent attempts to get money without working. But even if it were, SSDI would still only explain about one-quarter of the decline in the male participation rate over that time. There are many good reasons to reform disability insurance. But it’s not the singular driving force behind the decline of working men.

3) Do stay-at-home dads account for the change?

This is an easy one: no.

First, married men are more likely to work than non-married men. Second, fathers have stayed in the workforce more than non-fathers. Third, more than 75 percent of prime-age men not in the workforce do not have a working spouse.  Fourth, time-use surveys of non-working men have found that they are less likely to be caring for household members than working men. They do, however, watch more than twice as much television.

4) Are they in prison?

This question requires the most complicated answer. Technically, the 1.1 million prime-age men in prison aren’t hurting the participation rate among men, because they aren’t being counted as participants or non-participants. The government omits prisoners before it makes labor-force calculations.

But that’s not even close to the full story. The more meaningful and troubling statistic is that about 9 million prime-age men have been incarcerated. “These men are substantially more likely to experience joblessness after they are released from prison,” the White House economists reported, “and in many states [they] are legally barred from a significant number of jobs.”

There have been several attempts to measure the cost of prison on subsequent employment. A 2010 analysis by Pew Charitable Trusts found that incarceration reduced the average work time of a typical 45-year-old man by 19 percent. Another study estimated that people who go to prison are 30 percent less likely to subsequently find a job than a non-incarcerated person of their age.

There is no question that it’s harder to find a job after somebody gets out of prison. But even if one makes the most aggressive assumptions about the cost of incarceration to unemployment, America’s prison and ex-prison population only explains a fraction of the disappearing male workforce.

It is conceivable that, by degrees, each of these variables is eating away at prime-age male participation rates. Men who leave prison during a time of historic incarceration have had a hard time finding steady work; some men have turned to disability payments to disengage from a workforce that offered poor pay; some adults have returned to school; and a few (perhaps very few) fathers are staying home while their spouses work.

But behind all of these trends, there is a larger story: the decline of sectors dominated by male workers. In 1954, the highwater mark for male participation, the manufacturing and construction sectors accounted for nearly 40 percent of all jobs. Now, after the long decline of manufacturing and the end of the housing bubble, they account for just 13 percent. These are jobs that men without a college degree can count on, and they're much rarer than they used to be. The White House report notes that "when the share of state employment attributable to construction, mining and to a lesser extent manufacturing are higher, more prime-age men participate in the labor force.” In other words, men are more likely to work in areas where the state directly subsidizes employment in male-heavy occupations.

But the private sector is shifting toward work that has historically been done by women. There are four occupations expected to add more than 100,000 jobs in the next decade: personal care aides, home health aides, medical secretaries, and marketing specialists. All of them currently have more female workers than male. "Some of the decline in work among young men is a mismatch between aspirations and identity," Lawrence Katz, a professor of economics at Harvard University, told me last year. "Taking a job as a health technician has the connotation as a feminized job. The growth has been in jobs that have been considered women’s jobs—education, health, government."

Perhaps the United States needs some sort of massive national building project to put these men back to work in jobs that they would be proud and willing to do. The promises of Donald Trump to the contrary, the United States is not poised to bring back manufacturing jobs.

But what about construction jobs?

Several weeks ago, Conor Sen, a portfolio manager and a columnist at Bloomberg View, wrote a widely shared essay predicting that housing would become the dominant economic story of the next five years. Sen worried that it would be hard to find enough workers (mostly men, since construction employment is about 90 percent male) to build the requisite number of houses. After all, construction skews young and less educated, and the U.S. is getting older and more educated. "If we had to find 500,000 construction workers tomorrow, from a math standpoint it would be impossible," he wrote. "The slack isn’t there.”

But the slack is there. Millions of able-bodied men have dropped out of the labor force, mostly because they have stopped looking for work. Many of them badly need to leave their neighborhoods in Appalachia, the Rust Belt, and the Deep South, where the rate of non-working men often hovers around 40 percent. Meanwhile, the U.S. needs more affordable housing construction, particularly in its richest and most populous metro areas.

The White House report lists several ways to raise the employment level of male workers, like criminal justice reform and removing occupational licenses. But here is a bolder plan: A state-and-federally funded voucher program that moves men from economically stricken areas toward metros in need of construction workers.

This plan would require heroic (and heretofore unprecedented) participation from all levels of government. Metropolitan areas might have to rezone neighborhoods to provide for more low-income housing, and the GOP Congress would have to agree to what is essentially a multi-billion dollar stimulus package, which it has repeatedly said it will never do.

Although abrupt moves can be especially socially disruptive for families with young children, the non-working male population skews young, single, and childless. They are already unmoored, untethered from the economy. In this case, encouraging men to dislocate might move them closer to a job and social network that would bring them meaning and community, above and beyond the obvious benefits of a steady paycheck. This could be a way to bring millions of men back from the brink.