To understand why, begin with two crucial words. The first is "marriage." Group love (sometimes called polyamory) is already legal, and some people freely practice it. Polygamy asserts not a right to love several others but a right to marry them all. Because a marriage license is a state grant, polygamy is a matter of public policy, not just of personal preference.
The second crucial word is "polygyny." Unlike gay marriage, polygamy has been a common form of marriage since at least biblical times, and probably long before. In his 1994 book The Moral Animal: The New Science of Evolutionary Psychology, Robert Wright notes that a "huge majority" of the human societies for which anthropologists have data have been polygamous. Virtually all of those have been polygynous: that is, one husband, multiple wives. Polyandry (one wife, many husbands) is vanishingly rare. The real-world practice of polygamy seems to flow from men's desire to marry all the women they can have children with.
Moreover, in America today the main constituents for polygamous marriage are Mormons and, as Newsweek reports, "a growing number of evangelical Christian and Muslim polygamists." These religious groups practice polygyny, not polyandry. Thus, in light of current American politics as well as copious anthropological experience, any responsible planner must assume that if polygamy were legalized, polygynous marriages would outnumber polyandrous ones—probably vastly.
Here is something else to consider: As far as I've been able to determine, no polygamous society has ever been a true liberal democracy, in anything like the modern sense. As societies move away from hierarchy and toward equal opportunity, they leave polygamy behind. They monogamize as they modernize. That may be a coincidence, but it seems more likely to be a logical outgrowth of the arithmetic of polygamy.
Other things being equal (and, to a good first approximation, they are), when one man marries two women, some other man marries no woman. When one man marries three women, two other men don't marry. When one man marries four women, three other men don't marry. Monogamy gives everyone a shot at marriage. Polygyny, by contrast, is a zero-sum game that skews the marriage market so that some men marry at the expense of others.
For the individuals affected, losing the opportunity to marry is a grave, even devastating, deprivation. (Just ask a gay American.) But the effects are still worse at the social level. Sexual imbalance in the marriage market has no good social consequences and many grim ones.
Two political scientists, Valerie M. Hudson and Andrea M. den Boer, ponder those consequences in their 2004 book Bare Branches: Security Implications of Asia's Surplus Male Population. Summarizing their findings in a Washington Post article, they write: "Scarcity of women leads to a situation in which men with advantages—money, skills, education—will marry, but men without such advantages—poor, unskilled, illiterate—will not. A permanent subclass of bare branches [unmarriageable men] from the lowest socioeconomic classes is created. In China and India, for example, by the year 2020 bare branches will make up 12 to 15 percent of the young adult male population."