In 2011, Michael Bloomberg gave the Sierra Club $50 million, the biggest donation in the club’s history, to expand the organization’s Beyond Coal initiative, which sought to shut down coal-fired power plants across the country. He gave $30 million more in 2015. The Beyond Coal campaign now says it has helped shut down 251 coal-burning plants, thanks in large part to Bloomberg’s donations.

The campaign’s success is good news if you’re an environmentalist who wants to replace coal with renewable energy, but not so great if you’re a coal miner watching his livelihood slip away. In a democracy, both sides might argue the issue of whether to shut down coal-fired power plants to their legislators or express their preference at the voting booth. But Bloomberg’s $80 million in donations inarguably gave a boost to the environmentalists’ side, and led to changes on the ground.

Bloomberg’s gift is not unusual in an era in which extreme wealth is growing in America. Individuals made rich by the tech boom or the financial markets are vowing to give money away at unprecedented levels. But this type of philanthropy creates challenges in a democracy, argues David Callahan, the founder and editor of the website Inside Philanthropy, in his new book, The Givers: Money, Power, and Philanthropy in a New Gilded Age. The gifts come at a time when government is shrinking, and when, in some cases, philanthropic dollars replace or supplant government functions. That can mean that it’s philanthropists who decide what scientific issues are researched, what types of schools exist in communities, and what initiatives get on ballots. “It’s great to have these new donors appearing on the scene at a time when government is being cut,” Callahan told me, in a phone interview. “On the other hand, there’s no question that with money comes power and influence.”

Philanthropists have long experimented with new models that government would soon adopt—the Robert Wood Johnson Foundation laid the groundwork for today’s 9-1-1 emergency response system, for example. What’s different today, Callahan argues, is that instead of slow-moving foundations dispensing the fortunes of long-dead Americans in cautious policy forays, today’s philanthropists are alive, and very much involved in how their money is spent. They want to see rapid change occur in their lifetimes, and they often have very specific ideas about what that change should look like. These ideas often relate to re-shaping policy in the city—and country—where they live.

As the founder and editor of a news site about philanthropy, Callahan is intimately familiar with today’s living donors and how they are spending their money, and his book is replete with examples of philanthropists that are upending the democratic process. The husband-and-wife team of Barry Diller and Diane von Furstenberg orchestrated a process to build a new island park in the Hudson River, donating the bulk of the money needed, without seeking input from locals, he writes. The philanthropist David Welch financed a nonprofit, Students Matter, which initiated a lawsuit, Vergara v. California that sought to get rid of California laws that made it hard to get rid of tenured teachers. The Walton Family Foundation teamed up with the Broad Foundation and others in an effort to move over half of all schoolchildren in Los Angeles to charter schools. Each of these initiatives had opponents who had less sway than the philanthropists in decision-making, and who were, in some cases, left out entirely of the democratic process. “We face a future in which private donors—who are accountable to no one—may often wield more influence than elected public officials, who (in theory, anyway) are accountable to all of us,” he writes.

There’s long been a debate in academic circles about what role foundations should have in democracies. In an essay in the book Philanthropy in Democratic Societies, published last year, the Stanford sociologists Aaron Horvath and Walter W. Powell argue that a particular form of new philanthropy, which they call “disruptive philanthropy,” can be harmful. They define disruptive philanthropy as philanthropy that competes with the government to provide services, rather than collaborates with it, Horvath told me, in a phone call. The trouble is that this type of giving can reshape the public agenda. “Disruptive philanthropy seeks to shape civic values in the image of funders’ interests and, in lieu of soliciting public input, seeks to influence or change public opinion and demand,” Horvath and Powell write. Charter schools are a prime example—they are often funded to compete with the public-school system, and eventually replace it. Often, these gifts are made with little public oversight or input, and yet make huge changes to public policy.

This argument that philanthropists challenge democracies isn’t necessarily new. When John Rockefeller first went to Congress in 1912 to get a federal charter to establish a foundation, he encountered resistance from government leaders who found the whole idea of philanthropy suspect, as Rob Reich of Stanford (not to be confused with Robert Reich, the former Labor Secretary) wrote last year in PS, the journal of the American Political Science Association. At the time, the chair of the ACLU testified that Rockefeller’s foundation was “repugnant to the whole idea of a democratic society.”

Callahan isn’t sure where he stands on this debate. As he points out, it’s hard to argue against individuals providing nets to prevent the spread of malaria, or giving away all their money before they die. But he makes a convincing argument that philanthropists are more aggressive than ever, and that this is cause for concern.

Philanthropists have given rise to think tanks like the Heritage Foundation and the Center for American Progress, which publish research that bolsters certain policy positions. For instance, The New Yorker’s Jane Mayer recently wrote about about Robert Mercer, the philanthropist whose foundation has supported nonprofits and think tanks that consistently attacked Hillary Clinton, who went on to lose the presidential election. They have transformed the physical landscape of cities such as Tulsa and Los Angeles, building parks that their foundations pay to maintain and museums named after themselves. They have donated millions to expand charter schools in Los Angeles, where they hope to enroll half of all students in the next decade. Callahan says that one of his main goals in writing the book was to make people more aware of just how influential today’s philanthropists are. “I think the more people understand how much power the wealthy have through philanthropy, the more they’re likely to see it as part of this larger pattern of the wealthy speaking with a larger and larger voice, even as ordinary people struggle to be heard at all,” he told me.

Callahan does have some thoughts about how to push back against the increasing power of wealthy donors in civic society. One idea is changing the tax status of some organizations. After all, some philanthropists use the groups they fund to advocate for public policy changes, but are not taxed on those donations. For instance, the Einhorn Family Foundation, a group run by a Wisconsin venture capitalist, paid to put billboards in minority neighborhoods that read, “Voter Fraud is Felony!” and reminded people that it could be punishable by up to three years in prison. Though such billboards are arguably partisan because of where they were located, the Einhorn Family Foundation is tax-exempt, Callahan writes. The public is subsidizing some of these donations through charitable tax breaks, which will cost the U.S. government around $740 billion over the next decade.

Many of the groups created by foundations to work on public-policy issues are 501(c)3s, and donations to organizations that are 501(c)3s are tax-exempt. Donations to organizations that are 501(c)4s—groups whose goals include influencing legislation or the general public—are not tax-exempt. Yet many 501(c)3s still undertake efforts that arguably seek to influence the general public—writing reports that argue for or against minimum-wage rules, for example. The IRS should more narrowly define what constitutes charitable activities by 501(c)3 groups, Callahan argues, so that money given to such organizations, if used for partisan purposes, is no longer tax-deductible, and taxpayers are no longer subsidizing philanthropists’ public-policy plays.

Callahan also argues for making foundations more accountable, so that ordinary citizens know when philanthropic money is being spent to try to influence policy and their lives. Nonprofits advocating for changes on the ground can do so without revealing their donors, Callahan writes. Foundations should publish more information about where their money is going, he argues, and the IRS should bolster its division tasked with tax-law compliance. Though such enforcement costs money, it could be paid for by the two-percent federal tax that foundations pay on their annual investment income, which is not currently earmarked for that purpose. Callahan also suggests the creation of a new office of charitable affairs that could also assist with compliance.

Of course, these policies may be a tough sell in an era where many Americans distrust government. That’s why entrusting philanthropists themselves with changing their ways might be the best solution. In The Givers, Callahan documents a few examples of organizations that are seeking feedback from the communities where they work, rather than subverting them. The Chicago Community Trust, for example, invites Chicagoland residents to come and talk to them about the pressing problems they see, which then informs the Trust’s giving priorities. The intended beneficiaries—low-income women—run a foundation in Boston created by a woman named Karen Pittleman. But these examples are few and far between.

Philanthropists may not heed Callahan’s advice; part of the benefit of having so much money to give away is doing what you want with it without listening to anybody else. But in a society where the haves are increasingly separated from the have-nots, some wise philanthropists may begin to understand that there’s a problem when having money means not just improving your own life, but deciding what happens in other people’s lives too.