Long before the Supreme Court legalized same-sex marriage in the United States, a somewhat surprising group of people became the most formidable legal opponents of gay marriage: cake bakers. Along with photographers, florists, and other vendors who typically provide services for weddings, a handful of bakers across the country claimed they shouldn’t have to serve gay ceremonies; doing so, they said, would violate their religious beliefs.

In the half decade or so since these claims started coming up, the bakers have mostly lost, and on Thursday, that losing streak continued. The Colorado Court of Appeals ruled that the owners of Masterpiece Cakeshop in Lakewood, Colorado, violated the state’s public-accommodations law when its owner, Jack Phillips, refused to make a cake for Charlie Craig and David Mullins, a gay couple who wanted to marry in 2012. “Phillips believes that decorating cakes is a form of art, that he can honor God through his artistic talents, and that he would displease God by creating cakes for same-sex marriages,” the court wrote in its decision. Even so, it found, a lower court had the right to issue a cease-and-desist order against Masterpiece: Having to bake a cake for a gay wedding doesn’t place an undue burden on Philips’s religious exercise, nor is it a violation of his right to free speech. It's possible that Masterpiece will appeal the case to the Colorado Supreme Court. In a statement, the group that’s representing him, the Alliance Defending Freedom, said that it is discussing “further legal options.”

Legally, this case has a few interesting aspects. First, the court of appeals takes up a somewhat-common free-speech argument, that cake baking is a form of art, and thus a form of free speech. Not so much, the court said. “The act of designing and selling a wedding cake to all customers free of discrimination does not convey a celebratory message about same-sex weddings,” the justices wrote. “To the extent that the public infers from a Masterpiece wedding cake a message celebrating same-sex marriage, that message is more likely to be attributed to the customer than to Masterpiece.”

Under the law, there’s an important distinction between refusing to provide a service and refusing to “say” something. As Doug NeJaime, a  UCLA law professor, put it to me in July, “It’s the difference between not providing service because the person’s gay and not doing something in particular because it’s particular speech.” This has been an important factor in other, similar cases, so it’s significant that the Colorado court rejected the idea that cake-baking is primarily a form of artistic expression and speech.

State laws also play an important role in this case. As the justices wrote, “Masterpiece violated Colorado’s public accommodations law by refusing to create a wedding cake for Craig’s and Mullins’ same-sex wedding celebration.” Colorado is one of 22 states, along with the District of Columbia, that have laws forbidding discrimination on the basis of sexual orientation in areas like housing, employment, and public accommodations—basically, any business that sells something to people. It’s also not one of the 21 states that has a version of the Religious Freedom Restoration Act on the books, which provides special protections for religious groups who claim that certain laws place a burden on their ability to practice their religion. Although the court of appeals considered whether Colorado’s law placed a burden on Phillips’s conscience, it specifically noted that the state doesn’t have a law requiring religious exemptions from rules such as public-accommodations laws.

Although the Masterpiece case follows the pattern of rulings against religious cake bakers, florists, and photographers, the pattern itself is somewhat curious: Why is it that these religious-freedom claims aren’t coming up more frequently in the South, for example, where support for gay marriage is still a minority viewpoint? In part, it’s because these states have different legal priorities from places like Colorado and Oregon, where cake bakers have been shut down in high-profile cases. Most Southern states have special protections for religious groups, but they don’t have special protections against LGBT discrimination. Bakers in Colorado have to bake for gays, and if they don’t, they can be taken to court. But in the majority of states in America, same-sex couples don’t really have the legal cover to put up a fight.