Six songs into his new album, Kendrick Lamar locks himself in a hotel room and faces a brutal assault from his own conscience. His voice shredded like that of a man dying in the desert, with horns in the background wailing like a pack of dogs, Lamar raps that his talent’s worthless, that he’s sold out his friends, that his art has changed nothing, and that success has made him complacent. Thoughts of suicide flash through his mind, and the maid’s knocks on the door go unanswered.

The chorus to this horror-show of a song, “u,” reads like this: “Loving you / is complicated.”

Complicated—not impossible, not difficult, but complicated. Everything in Lamar’s world is complicated, probably because everything in the real world is. Whereas so much of pop culture, pop music, and hip hop filters life to seem slightly slicker and simpler, the 27-year-old Compton rapper adjusts the contrast settings, highlighting the dark corners and the cracks. He’s the only major rapper now working who fully embraces the form’s discursive potential, chasing every contradiction he comes across to its sometimes-ugly ends.

His new full-length record, To Pimp a Butterfly, comes after two albums of richly detailed narratives about growing up poor in Compton. But 2012’s Good Kid, M.A.A.D. City went platinum, and his lyrics now focus not on who he once was but who he is: a celebrity. Yes, Butterfly is part of the not-always-so-grand tradition of art about how hard it is to be famous. But it’s also a dense, astonishing, cliché-busting confession about how his fear of failure intertwines with his race, community, ambition, and psychology.

The experimental musician Flying Lotus produced the opening track, “Wesley’s Theory,” and his signature sound—jazz instrumentation and hip-hop layered into chaotic collages—is all over the album. The unsettled vibe suits the words, in which every moment of pride or assurance or even righteous rage comes with an asterisk. For example, the first verse of “Wesley’s Theory” sees Lamar reminisce about his lust for a record deal; in the second verse, though, Uncle Sam entices Lamar to use his new cash to go on a shopping spree—and then promises to hit Lamar with tax-evasion charges like those that derailed Wesley Snipes's career. The message: America will try to sabotage a black person's success.

Similar anxieties pervade the album, but Lamar still can have fun. “King Kunta” is a brag track like no other, sounding like Parliament Funkadelic covering “Smooth Criminal” (which is, in fact, sampled) as Lamar imagines the Roots slave Kunta Kinte turned into a rap boss. “I” is the flipside of “u,” featuring a perky Isley Brothers sample and Lamar vowing to love himself. The interlude "For Free?" allows Lamar to reassert himself as a virtuoso vocalist, jabbering crassly about his self worth over Whiplash-style drum insanity. And “These Walls” is a lush sex jam—until the final verse, when, in a quintessential Lamar twist, he feels guilt about his girl's incarcerated ex.

Released by surprise on Sunday night, the album has divided fans already. With its woozy arrangements and allergy to conventional song structures, it’s the least inviting work yet from an artist who was never interested in chasing the mainstream. Some people probably find Lamar’s savior complex grating; he compares himself to Nelson Mandela, stages a posthumous conversation with 2Pac, and at one point, suggests that the solution to gang violence is more respect between men. And he's working in a lyrical realm far more abstract than those he's inhabited before, squaring off against "Lucy"—Lucifer, tempting him to become a hack rapper at every turn—rather than the flesh-and-blood characters who populated Good Kid, M.A.A.D. City and 2011's Section.80.

But his creativity or talent just can't be denied, and his eye for nuance makes the emotional payoffs on the album both harder-earned and more powerful. “How Much a Dollar Costs” might be the most stunning example, crossing Good Kid-style concrete storytelling with his large philosophical concerns over an achingly lovely piano progression. The narrative: A homeless man confronts Lamar for not giving him a dollar, and Lamar enters an emotional spiral, trying to justify his miserliness—"my selfishness is what got me here, who the fuck I'm kiddin'?" Then the homeless person reveals himself to be God. Whoops. Among other things, the transient deity tells Lamar, "your potential is bittersweet," offering up a beautifully simple description for Lamar's beautifully complex concerns.