If there was a theme to Ricky Gervais’s performance at Sunday’s Golden Globe Awards, it was that pop culture’s purported “transgender moment” is awfully funny. He pretended that he thought Eddie Redmayne, who plays a transgender woman in The Danish Girl, was actually female. He speculated about what Jeffrey Tambor does with his testicles on Transparent. And toward the beginning of his monologue, Gervais said that he himself had changed a lot in a year, “but not as much as Bruce Jenner, obviously.”

There was a short pause. Some laughter from the audience. Then a follow-up joke.

“She became a role model for trans people everywhere, showing great bravery in breaking down barriers and destroying stereotypes. She didn’t do a lot for women drivers, but you can’t have everything, can ya?”

Some online reactions called Gervais’s routine “transphobic,” and on Tuesday, Gervais responded. “Suggesting a joke about Caitlin Jenner is automatically transphobic is like suggesting a joke about Bill Cosby is automatically racist,” he wrote on Twitter. A few hours later: “I made a joke about Caitlyn Jenner killing someone in her car. I’m .”

Gervais is right that it’s silly to label every joke about a trans person as “transphobic.” He also has a pretty good case that the car joke wasn’t hurtful to trans people. It was hurtful to Jenner’s reputation, perhaps (though it’s not like the fatal car crash at issue went unpublicized), and also to anyone who suspects that a good way to help end stereotypes about women is to stop making jokes that rely on them.

Yet Gervais’s defense misses the point of the annoyance he sparked (yes, annoyance: Many comedians who face any sort of criticism like to say they’ve offended and outraged entire nations, but that often gives them too much credit). The regressive part of his Jenner routine was in the first joke, the one where he played for laughs the mere fact that Bruce was now called Caitlin.

That’s fundamentally the same joke he made about Tambor and Redmayne. There’s not a lot to it. Its content, simply, is this: It’s funny when people previously seen as men change themselves to be seen as women.

This supposed punchline, as should become clear to anyone who thinks about it for a moment, results from and feeds the stigmas that trans people face. The idea that presenting as something other than the gender you were told you had at birth is a fundamentally hilarious proposition—whether because of the methods involved or because of the intentions behind doing so—is a very obvious sign of devaluation. The very work of Jenner, Tambor, and Redmayne, not to mention the many transgender writers and advocates across media, is in part about explaining and combating this fact.

Gervais’s joke is also among the most elemental joke kinds there are: the joke of difference. It stems from the idea that “the other”—whatever the other may be to the speaker and the audience—is inherently freaky. It accounts for part (and only part) of the perceived humor in blackface, or mocking accents, or any number of other worn-out tropes that routinely upset swaths of the population just trying to live their lives. It is also usually not a very good joke, both because people have heard it so many times and because it reinforces something that’s definitionally boring—the status quo.