There's a track on Ashlee Simpson's debut record Autobiography titled "Shadow," and listening to it now, America really should have known better. “I was stuck inside a broken life I couldn't wish away / She was beautiful / She had everything and more / And my escape was hiding out and running for the door,” she sings in the song’s first verse. The “she” in question was her sister Jessica, to whom the titular shadow belonged. “Shadow,” like most of the songs on Autobiography, was more somber than the more famous Simpson sister’s music. It was personal, a plea to let the kid sis with the dyed black hair her be freed from expectations created through connection to her blonde, more famous sister.

Autobiography was released in July 2004 to strong sales—just under 400,000 copies sold the first week, with a clear debut atop the Billboard Hot 100. She parlayed her MTV series The Ashlee Simpson Show with great success, a stark contrast to how Jessica’s Newlyweds with then-husband Nick Lachey failed to boost third album In This Skin’s sales above 64,000 in its first week. Sure, one could argue Ashlee’s show was only successful because of Newlyweds’ growing audience—and the increasing fame of Jessica—but this was still a victory for the younger Simpson. She’d sold as many copies of her album in a week than Jessica did in several months. Ashlee had taken the first important step beyond living in that shadow.

On October 23, 2004, ten years ago this week, Ashlee took yet another step. Unfortunately, this one was more of a jig, meant to distract from the fact that she was just caught lip-syncing on Saturday Night Live. And with that step fell the first domino in a line of misfortunes for her family.

Watching the clip above, it's hard not to experience secondhand embarrassment. The lip sync is just cringeworthy now as it was then. Having performed—er, "performed"—her hit single "Pieces of Me" earlier in the show, Ashlee was set to do "Autobiography." However, the vocal track that played instead was for "Pieces." No sound came out of her mouth. She did a feeble dance to cover up the mistake, but the damage was done. In an instant, Ashlee had made sure she would not just be known as Jessica's little sister any more. She'd be known as the SNL lip-sync girl.

Ashlee blamed acid reflux for the backing track, but the damage was done. SNL grand poobah Lorne Michaels wagged his finger at Ashlee publicly, and even in the pre-Twitter-and-thinkpiece-overload days, she got pummeled by press. The Washington Post columnist Lisa de Moraes went so far as to compare it to Janet Jackson's wardrobe malfunction at the Super Bowl earlier that year. Ashlee had trouble recovering—most notably suffering a booing crowd after a very live and very off-key rendition of her song "La La" at the 2005 Orange Bowl.

Though she released another album the next year—with a return appearance on SNL that went much better—she never quite recovered. I Am Me sold a third of the number of copies Autobiography sold, not even reaching a million at a time when almost every pop star could still pass that benchmark. Fans turned from rooting for Ashlee to gleefully tearing into her. But in hindsight, it seems almost inevitable that America would have turned viciously on a member of the Simpson clan.

This was a family made very famous because of Jessica's persona: the stereotype of a dumb blonde. The years since have proven this characterization to be false, but fact remains that Newlyweds became a phenomenon when Jessica had to ask if "Chicken By The Sea" was fish or chicken. The Simpsons were there to be laughed with, until it was time to laugh at them. Which made them, at that particular moment in celebrity culture—celebreality, as VH1 branded it—somehow seem more authentic. If Jessica was a manufactured star, at least she was one that Americans felt like they knew. So the Newlyweds ratings were massive.

Though Autobiography is a better pop album than its place in history would indicate, Ashlee came off as more prefab: a second version of the Simpson brand, but edgy and with dark hair! She was obviously supposed to be the anti-Jessica who people who liked Jessica could still love. Even the steps taken to make her a star—giving her an MTV show of her own being paramount—felt like a retread. What worked for Jessica must work for Ashlee, right?

After her fall, Ashlee could best be compared to a later up-and-coming (at the time) performer who experienced disaster on the Studio 8H stage: Lana Del Rey. In both cases, you've got a dark, moody singer with industry connections who got unmercifully ripped for her poor performance. Never mind that plenty of other artists have bombed on SNL (remember Karmin?). No, because they felt so manufactured, Ashlee and Lana were easy targets.

But while Lana worked the backlash into her shtick—odd and affected works for her—Ashlee's disaster revealed her to be a patchwork pop star. Unlike her sister, she was no longer relatable. She was a product of the machine. Maybe that's why Ashlee's album sales cratered on her sophomore effort; when the wrong track played on SNL, America saw the seams. And they didn't like what they saw.

Yet the SNL debacle didn't only hurt Ashlee's success. Not long after, the Simpson dynasty began to crumble. Correlation does not prove causation, of course ... and yet there's something remarkable about how quickly things fell apart for the Simpson clan. Exactly one year to the month after SNL, Jessica announced her separation from—and subsequently divorced—Nick Lachey. After the divorce, she started dating Dallas Cowboys quarterback Tony Romo and was promptly blamed in the press for his poor play. She earned the nickname Yoko Romo.

Even worse off was Joe Simpson, the girls' father. Always a bit of an odd character, having made inappropriate comments about his daughter's breasts that Jessica had to refute, Joe and his wife Tina split in 2012. In the years since, Joe's been dogged by rumors that he's gay and dating a much younger man—a potentially big reversal for the once-youth pastor. (Joe's denied the rumors publicly.)

Meanwhile, Ashlee's third album, Bittersweet World, was released in 2008 to ghastly sales and only slightly better reviews. She dated a string of men, marrying Fall Out Boy's Pete Wentz and divorcing him after three years. She's now married to Diana Ross' son Evan. Ashlee still makes celebrity news headlines—though often in relation to her sister. That's because, after her spate of bad luck, a funny thing happened to Jessica. She made quite the comeback with the Jessica Simpson Collection, a line of shoes, clothes, fragrances and more. One might dismiss the collection as a trifle of a celebrity vanity line on instinct, but it's actually valued at a billion dollars. Jessica made celebrity work for her—and long after her music career failed to truly ignite, she remains famous.

In the end, neither Ashlee nor Jessica became the massive pop star they wanted to be. The House of Simpson, once seemingly unbreakable, fell apart after Ashlee's lip sync—and, of course as celebreality fell out of style. In its place, the older sister built the House of Jessica. And the younger sister—the girl who lip-synced on Saturday Night Live—was once again relegated to the shadows.