Some of the wealthiest, most-educated towns in the United States have the biggest academic-achievement gaps between white students and their peers of color. That is one of the depressing facts emerging from a wide-ranging new analysis of more than 200 million test scores of 40 million students from around the country between 2009 and 2013 by Stanford University researchers.

Comparing district-level data across states is complicated because not all students take the same tests. The researchers created a database that allows these comparisons, providing what they say is the most in-depth look at academic disparities across the country. They found wide disparities in prosperous university towns like Berkeley, California, Chapel Hill, North Carolina, and Evanston, Illinois—cities often heralded for driving economic development and opportunity. On average, black students score two grade levels lower than white students in their districts, while Latinos score one-and-a-half grade levels lower. The most and least socioeconomically disadvantaged districts are four grade levels apart.

The gaps emerge both between low-income and affluent children, and between white children and their peers of color. The researchers suggest that this is largely because black and Latino students are more likely to come from poor families, and to attend high-poverty schools. But even when students share similar socioeconomic backgrounds and attend similar schools, white students fare better. Although the study does not offer much to explain why this is so, Sean Reardon, a Stanford education professor and one of the leads on the study, hypothesized during a phone interview it may be because schools track white students into more-rigorous courses. This hypothesis tracks with other recent findings, such as a report from two Vanderbilt University researchers suggesting schools underestimate the abilities of black and Latino students, and a study from Johns Hopkins University that found that white teachers are less likely than black teachers to think their black students will graduate from high school.

Beyond the potential effects of tracking, children from wealthy families seem to benefit from many variables outside the classroom, including enriching home environments, safe neighborhoods, good childcare, after-school activities, and the education level of their parents. Highly educated parents in these towns seem to have a “heightened focus” on education, Reardon said, and they are increasingly willing to spend resources to ensure that their children are academically successful. While less-educated parents certainly want their own children to do well, they tend to have less disposable income. Because white parents are more likely than parents of color to be highly educated and to earn more, white children are more likely than their peers of color to have access to enriching educational experiences outside of the classroom.

“I’m not really surprised,” said Sonja Brookins Santelises, the vice president of K-12 policy and practice at the nonprofit Education Trust and the former chief academic officer for Baltimore City Public Schools. Families with means, she pointed out, are spending more than they used to on experiences outside the classroom that can increase test scores for their kids. This leaves children from lower-income families struggling to catch up.

A child who grows up in an affluent home with professors for parents in Berkeley, for instance, is more likely to be enrolled in activities like ballet and soccer than a child in Berkeley whose parents don’t have as many resources. According to the Pew Research Center, 84 percent of families who earn at least $75,000 say their children are involved in sports, and 62 percent say their kids take music, dance, or art. For families earning less than $30,000, those participation rates fall to 59 percent and 41 percent, respectively. The child from the affluent family is also more likely to have parents who can assist with homework and who have the financial ability and time to take trips to museums or galleries on weekends—out-of-school activities that research suggests boost in-school test scores.

Studies suggest that music lessons can help with everything from reading to spatial reasoning, and the College Board found that students who took art and music throughout high school scored higher on the SATs than students who did not.  A research paper from the National Endowment for the Arts suggested that eighth graders who were engaged in the arts throughout elementary school tested better in science and writing than their peers who were not involved in the arts.

While the recession and the arrival of No Child Left Behind prompted many schools to scale back such activities to focus on the subjects students are evaluated on, Brookins Santelises says the cuts have damaged student engagement, and likely widened achievement gaps in the process. Her daughters learned more about the American Revolutionary War during a short family trip to Philadelphia than they did in a classroom lesson on the topic, she said. But not all kids have parents who can take them to visit Independence Hall. For that reason, Brookins Santelises said, it’s more important than ever for schools with increasingly diverse student bodies to consider whether they’re offering students a range of learning experiences, from music to museum field trips, to expose children to lessons they might not get at home and to spark excitement about learning. She thinks doing so can have a positive impact on scores, but said that a recent emphasis on testing in schools has some educators zeroing in on test prep at the expense of field trips and art lessons. “What we end up doing,” she said, “is giving kids who are living in poverty the most impoverished learning experiences.”

Beyond that, though, Reardon’s research brings up the persistent issue of school segregation, which continues to have a profound impact on student achievement. “The results of these descriptive analyses are unequivocal. Racial segregation is strongly associated with racial achievement gaps; and the racial difference in the proportion of students’ schoolmates who are poor is the key dimension of segregation driving this association,” he writes in a paper accompanying the data. In short, racial segregation seems to lead to an unequal distribution of resources that is disproportionately harming poor children of color.

Kids of color are more likely to be poor than their white peers, and to attend high-poverty schools. High-poverty schools are more likely to have less-qualified teachers, weak parent-teacher networks, and they are less likely to offer advanced courses. Just 68 of the 1,000 poorest districts in the country have test scores at or above the national average, Reardon and his colleagues found, while only 16 of the 1,000 wealthiest districts have scores below average. “In other words,” Reardon writes, “we have little evidence that we know how to provide adequate educational opportunities for children growing up in low‐income communities.”

Although outright segregation is illegal, many schools remain largely segregated because neighborhoods are similarly divided or because school-boundary lines have been drawn in ways that keep them so. There is evidence that taking deliberate steps to reduce both economic and racial segregation in schools raises the scores of children of color, but it requires communities to take deliberate steps toward integration in both housing and schooling, and it’s complicated by the fact that sometimes even children at the exact same school have very different experiences because of outside forces. Consider the experience of a kid whose parent has the wherewithal to ask the principal to place her child with a particularly well-liked teacher. If that parent is an active volunteer that the school wants to keep happy, that may mean one less seat in the class for a kid whose parents can’t be as engaged. While no reasonable person wants to punish parents who participate, educators are also looking for ways to help children with less-involved parents who arguably need support at school more than children who show up to class with supportive networks already in place.

Reardon is blunt when he says there is no clear fix. “This isn’t a little problem,” he said. “We’ve got to work on lots of fronts at once.”