On a clear morning this past February, the inmates in the B Yard of Pelican Bay State Prison filed out of their cellblock a few at a time and let a cool, salty breeze blow across their bodies. Their home, the California prison system’s permanent address for its most hardened gangsters, is in Crescent City, on the edge of a redwood forest—about four miles from the Pacific Ocean in one direction and 20 miles from the Oregon border in the other. This is their yard time.

Most of the inmates belong to one of California’s six main prison gangs: Nuestra Familia, the Mexican Mafia, the Aryan Brotherhood, the Black Guerrilla Family, the Northern Structure, or the Nazi Lowriders (the last two are offshoots of Nuestra Familia and the Aryan Brotherhood, respectively). The inmates interact like volatile chemicals: if you open their cells in such a way as to put, say, a lone member of Nuestra Familia in a crowd of Mexican Mafia, the mix can explode violently. So the guards release them in a careful order.

“Now watch what they do,” says Christopher Acosta, a corrections officer with a shaved head who worked for 15 years as a front-line prison guard and now runs public relations for Pelican Bay. We are standing with our backs to a fence and can see everything.

At first, we seem to be watching a sullen but semi-random parade of terrifying men—heavily tattooed murderers, thieves, and drug dealers walking past one of five casual but alert guards. Some inmates, chosen for a strip search, drop their prison blues into little piles and then spin around, bare-assed, to be scrutinized. Once inspected, they dress and walk out into the yard to fill their lungs with oxygen after a long night in the stagnant air of the cellblock. The first Hispanic inmate to put his clothes on walks about 50 yards to a concrete picnic table, sits down, and waits. The first black inmate goes to a small workout area and stares out at the yard intently. A white guy walks directly to a third spot, closer to the basketball court. Another Hispanic claims another picnic table. Slowly it becomes obvious that they have been moving tactically: each has staked out a rallying point for his group and its affiliates.

Once each gang has achieved a critical mass—about five men—it sends off a pair of scouts. Two of the Hispanics at the original concrete picnic table begin a long, winding stroll. “They’ll walk around, get within earshot of the other groups, and try to figure out what’s going down on the yard,” Acosta says. “Then they can come back to their base and say who’s going to attack who, who’s selling what.”

Eventually, about 50 inmates are in the yard, and the guards have stepped back and congregated at their own rallying point, backs to the fence, with Acosta. The men’s movements around the yard are so smooth and organized, they seem coordinated by invisible traffic lights. And that’s a good thing. “There’s like 30 knives out there right now,” Acosta says. “Hidden up their rectums.”

A corrections officer at Pelican Bay conducts a search for contraband in an inmate's cell.

Understanding how prison gangs work is difficult: they conceal their activities and kill defectors who reveal their practices. This past summer, however, a 32-year-old academic named David Skarbek published The Social Order of the Underworld, his first book, which is the best attempt in a long while to explain the intricate organizational systems that make the gangs so formidable. His focus is the California prison system, which houses the second-largest inmate population in the country—about 135,600 people, slightly more than the population of Bellevue, Washington, split into facilities of a few thousand inmates apiece. With the possible exception of North Korea, the United States has a higher incarceration rate than any other nation, at one in 108 adults. (The national rate rose for 30 years before peaking, in 2008, at one in 99. Less crime and softer punishment for nonviolent crimes have caused the rate to decline since then.)

If your name is on a Bad News List, gang members attack you on sight—but remove your name when your debts are paid.

Skarbek’s primary claim is that the underlying order in California prisons comes from precisely what most of us would assume is the source of disorder: the major gangs, which are responsible for the vast majority of the trade in drugs and other contraband, including cellphones, behind bars. “Prison gangs end up providing governance in a brutal but effective way,” he says. “They impose responsibility on everyone, and in some ways the prisons run more smoothly because of them.” The gangs have business out on the streets, too, but their principal activity and authority resides in prisons, where other gangs are the main powers keeping them in check.

Skarbek is a native Californian and a lecturer in political economy at King’s College London. When I met him, on a sunny day on the Strand, in London, he was craving a taste of home. He suggested cheeseburgers and beer, which made our lunch American not only in topic of conversation but also in caloric consumption. Prison gangs do not exist in the United Kingdom, at least not with anything like the sophistication or reach of those in California or Texas, and in that respect Skarbek is like a botanist who studies desert wildflowers at a university in Norway.

Skarbek, whose most serious criminal offense to date is a moving violation, bases his conclusions on data crunches from prison systems (chiefly California’s, which has studied gangs in detail) and the accounts of inmates and corrections officers themselves. He is a treasury of horrifying anecdotes about human depravity—and ingenuity. There are few places other than a prison where men’s desires are more consistently thwarted, and where men whose desires are thwarted have so much time to think up creative ways to circumvent their obstacles.

Because he is a gentleman, Skarbek waited until we’d finished our burgers to illustrate some of that ingenuity. “How can you tell what type of cellphone an inmate uses,” he asked, “based on what’s in his cell?” He let me think for about two seconds before cheerily giving me the answer: you examine the bar of soap on the prisoner’s sink. The safest place for an inmate to store anything is in his rectum, and to keep the orifice supple and sized for the (contraband) phone, inmates have been known to whittle their bars of soap and tuck them away as a placeholder while their phones are in use. So a short and stubby bar means a durable old dumbphone; broad and flat means a BlackBerry or an iPhone. Pity the poor guy whose bar of soap is the size and shape of a Samsung Galaxy Note.

The prevalence of cellphones in the California prison system reveals just how loose a grip the authorities have on their inmates. In 2013, the California Department of Corrections and Rehabilitation confiscated 12,151 phones. A reasonable guess might be that this represented a tenth of all cellphones in the system, which means that almost every one of the state’s 135,600 inmates had a phone—all in violation of prison regulations. “Prison is set up so that most of the things a person wants to do are against the rules,” Skarbek says. “So to understand what’s really going on, you have to start by realizing that people are coming up with complicated ways to get around them.” Prison officials have long known that gangs are highly sophisticated organizations with carefully plotted strategies, business-development plans, bureaucracies, and even human-resources departments—all of which, Skarbek argues, lead not to chaos in the prison system but to order.

Craig Canary, an inmate in Pelican Bay’s Security Housing Unit, in his solitary-confinement cell

Skarbek trained in an economic school of thought known as rational-choice theory, which aims to explain human behavior as the product of reasonable decisions by economic actors. In many cases, rational-choice theory has shown behaviors to be rational that at first appear wild, irrational, or psychopathic. When people are encouraged or forced to act against their economic interest, they find work-arounds as surely as water blocked by a boulder in a stream finds a way to flow around it.

In 1968, one of the founders of rational-choice theory, Gary Becker, wrote a pioneering paper, “Crime and Punishment: An Economic Approach,” premised on the idea that the prevailing view of crime required revision. According to prior dogma, criminals were best understood as mental defectives, crazy people who couldn’t control their impulses. Becker, who won a Nobel in economics in 1992 and died this past May, suggested instead that criminals offend because they make careful calculations of the probability and likely cost of getting caught—and then determine that the gamble is worthwhile. This insight, Skarbek says, opened the study of crime up to economic theory.

Skarbek attended graduate school at George Mason University, a bastion of rational-choice theory. Its faculty is also friendly to unorthodox subject matter: Robin Hanson has published papers about using betting markets to augment democratic government, and has proposed that it is rational to freeze one’s head after death; Peter Leeson wrote The Invisible Hook, a 2009 account of the economics of piracy. Skarbek’s doctoral adviser, Peter Boettke, showed how the behavior of the Soviet economy actually made sense if you viewed it as controlled not just by the government but also by the black- and gray-market activities of citizens.

Prison, Skarbek claims, is the ultimate challenge for a rational-choice theorist: a place where control of the economic actors is nearly total, and where virtually any transaction requires the consent of the authorities. The Soviets had far less control over their people’s economic activity than prison wardens do over the few dollars available for prisoners’ commissary purchases. Both settings have given rise to alternate currencies and hidden markets. Most famously, cigarettes have become the medium of exchange in many prisons. But when they are banned, other currencies take their place. California inmates now use postage stamps.

A scene from general-population housing

Among the fundamental questions about prison gangs—known in California-corrections argot as “Security Threat Groups”—is why they arise in the first place. After all, as Skarbek notes, California had prisons for nearly a century before the first documented gang appeared. Some states don’t have prison gangs at all. New York has had street gangs for well over a century, but its first major prison gang didn’t form until the mid-1980s.

The explanation, Skarbek says, can be found in demographics, and in inmate memoirs and interviews. “Before prison gangs showed up,” he says, “you survived in prison by following something called ‘the convict code.’ ” Various recensions of the code exist, but they all reduce to a few short maxims that old-timers would share with first offenders soon after they arrived. “It was pretty simple,” he explains. “You mind your own business, you don’t rat on anyone, and you pretty much just try to avoid bothering or cheating other inmates.”

But starting in the 1950s, things changed: The total inmate population rose steeply, and prisons grew bigger, more ethnically and racially mixed, and more unpredictable in their types of inmate. Prisons faced a flood of first offenders, who tended to be young and male—and therefore less receptive to the advice of grizzled jailbirds. The norms that made prison life tolerable disappeared, and the authorities lost control. Prisoners banded together for self-protection—and later, for profit. The result was the first California prison gang.

That moment of gang genesis, Skarbek says, forced an arms race, in which different groups took turns demonstrating a willingness to inflict pain on others. The arms race has barely stopped, although the gangs have waxed and waned in relative power. (The Black Guerrilla Family has been weakened, prison authorities told me, because of leadership squabbles.) The Mexican Mafia was the sole Hispanic gang until 1965, when a group of inmates from Northern California formed Nuestra Familia to counter the influence of Hispanics from the south. Gang elders—called maestros—instruct the youngsters in gang history and keep the enmity alive.

What’s astonishing to outsiders, Skarbek says, is that many aspects of gang politics that appear to be sources of unresolvable hatred immediately dissipate if they threaten the stability of prison society. For example, consider the Aryan Brotherhood—a notoriously brutal organization whose members are often kept alone in cells because they tend to murder their cell mates. You can take the Brotherhood at its word when it declares itself a racist organization, and you can do the same with the Black Guerrilla Family, which preaches race war and calls for the violent overthrow of the government. But Skarbek says that at lights-out in some prisons, the leader of each gang will call out good night to his entire cellblock. The sole purpose of this exercise is for each gang leader to guarantee that his men will respect the night’s silence. If a white guy starts yelling and keeps everyone awake, the Aryan Brothers will discipline him to avoid having blacks or Hispanics attack one of their members. White power is one thing, but the need to keep order and get shut-eye is paramount.

Another common misconception about prison gangs is that they are simply street gangs that have been locked up. The story of their origins, however, is closer to the opposite: the Mexican Mafia, for example, was born at Deuel Vocational Institution, in Tracy, California, in 1956, and only later did that group, and others, become a presence on the streets. Today, the relation of the street to the cellblock is symbiotic. “The young guys on the street look to the gang members inside as role models,” says Charles Dangerfield, a former prison guard who now heads California’s Gang Task Force, in Sacramento. “Getting sentenced to prison is like being called up to the majors.”

But Skarbek says the prison gangs serve another function for street criminals. In a 2011 paper in American Political Science Review, he proposed that prison is a necessary enforcement mechanism for drug crime on the outside. If everyone in the criminal underworld will go to prison eventually, or has a close relationship with someone who will, and if everybody knows that gangs control the fate of all inmates, then criminals on the street will be afraid to cross gang members there, because at some point they, or someone they know, will have to pay on the inside. Under this model, prison gangs are the courts and sheriffs for people whose business is too shady to be able to count on justice from the usual sources. Using data from federal indictments of members of the Mexican Mafia, and other legal documents, Skarbek found that the control of prisons by gangs leads to smoother transactions in the outside criminal world.

Gangs effect this justice on the inside in part by circulating a “bad-news list,” or BNL. If your name is on a BNL, gang members are to attack you on sight—perhaps because you stole from an affiliate on the outside, or because you failed to repay a drug debt, or because you’re suspected of ratting someone out. Skarbek says one sign that the BNL is a rationally deployed tool, rather than just a haphazard vengeance mechanism, is that gangs are fastidious about removing names from the list when debts are paid.

An inmate of the Security Housing Unit at Pelican Bay is flanked by corrections officers as he is transported from one area of the unit to another.

No scholar writing in the law-abiding world, I was told by guards at Pelican Bay, can capture the reality of prison life in all its brutality. I was prepared for that to be true, even just based on my own reading. In 2005, Don Diva magazine interviewed a former guard at Rikers Island, who described the conditions of prison life in vivid terms. “[In each cell] you have a filthy toilet with no cover, a rusty sink, and a metal frame they call a bed,” he told the magazine. “Inmates use the toilet as a refrigerator in the summer to keep milk cool.” More vivid still was his description of inmate survival tactics:

Inmates are legendary for keeping razors in their mouths. Being able to “spit out a razor” is like a magic trick in jail. You could be in the mess hall, get into an altercation with another inmate, and the next thing you know he’s spit out two razors from both sides of his mouth and your face is slashed up … A nigga will become Houdini when it comes to survival. Spitting razors became such a problem that inmates immediately punched other inmates in the mouth as soon as an argument began. This was so that if the other inmate did have razors in his mouth, he would cut his own mouth up before even getting the opportunity to spit them out.

But I found that the staff at Pelican Bay had already been thinking about prisons the way Skarbek does. While I was there, Lieutenant Jeremy Frisk, the prison’s Institutional Gang Investigator, delivered a half-hour PowerPoint presentation focused on the managerial ingenuity of the gang leaders. One of the last slides featured a picture of the Chrysler chairman and 1980s business icon Lee Iacocca. “He was a very good manager,” Frisk said, “and turned Chrysler around from the brink of bankruptcy. And he could do that just from his management strategy: he never turned a wrench on a car, never assembled a door. But because of his ideas, they could make millions of dollars.” Frisk said gang leaders are the Lee Iacoccas of the prison world: brilliant managers of violence. (Since that presentation, I have found it impossible to look at a picture of Iacocca without imagining him stuffing his cheeks and rectum with razor blades.)

The safest place for an inmate to hide a phone is in his rectum. To keep the orifice sized and supple, inmates have been known to tuck a bar of soap away as a placeholder.

Pelican Bay opened in 1989 as an upgraded version of two famous old California prisons, San Quentin and Folsom, both of which still house inmates but function, as they always have, like enormous holding pens, hardly optimal for supervising a population of violent psychopaths who plot constantly to subvert the rules of the institution. Even the most secure housing at San Quentin, says Pelican Bay’s acting warden, Clark Ducart, was built so prisoners could all go from their cells to the yard together, with 50 men moving as an ungovernable mass. The walkways were narrow, and exposed prisoners to each other in ways that encouraged attacks. “As you walked guys to the shower,” he told me, “they’d get stabbed or speared.” Pelican Bay, by contrast, allows much greater levels of control, and a much more oppressive existence for anyone trying to plot a crime. The population is sectioned into yards and blocks that might have little contact with one another, and that allow the inmates to be managed with special attention to their gang affiliation. Upon identifying a gang member, the prison can modulate his location, freedom, and level of surveillance, to a degree that inmates have called stifling and inhumane.

On every cellblock at Pelican Bay, the guards post plastic identity cards on the wall, to keep track of which inmate is in which cell. These cards include each inmate’s name and photo. But the most-important information is conveyed by the cards’ color, which roughly correlates with probable gang affiliation: green for northern Hispanics, pink for southern Hispanics, blue for blacks, white for whites, and yellow for others, including American Indians, Mexican nationals, Laotians, and Eskimos. The information is crucial to the smooth running of the institution. Maintaining balance in a cellblock, and not putting a lone gang member in a situation where he might be surrounded by members of a rival gang, requires constant attention on the part of the corrections officers.

Out in the yard, when Acosta and I watched the inmates gather by gang, the guards knew exactly what was happening, and they could have intervened and broken up obvious gang activity. And it was obvious: nearly all gang members have gang tattoos across their torsos, and some have markings on their faces too. As Robert Mitchum growled in the remake of Cape Fear: “I don’t know whether to look at him or read him.”

Each interaction we observed between a correctional officer and a prisoner resembled bargain more than diktat. Before yard time finished, the guards let me inspect cells with them. The cells were livable, especially in comparison to the Rikers Island ones I had read about, even if the whole block had a dank locker-room smell. When I peeked in an inmate’s cell, I saw a dirty metal object in the sink. It was blunt and had a wire attached. “Stinger,” Acosta said. “Inmates use it to boil water. It’s illegal, but if the inmate isn’t doing anything wrong, a guard might let it pass.” He said that if a guard discovered a contraband item during an inspection, he might place it on the inmate’s bunk, just to show that he knew about it and could confiscate it at any time, if the inmate didn’t behave.

The guards asked inmates to show me a technique called “fishlining,” which involves attaching an object to one end of a string, sliding it out of a cell and into the hallway, and then using the other end of the string to yank it across the floor, this way and that, until it slides in front of the desired cell. A shatter-toothed Aryan Brother smiled at me and said he could send a book to an adjacent cell this way. (On his shelf: a single-volume edition of The Chronicles of Narnia and a Teach Yourself book on German.) The fishlines work as a way to distribute contraband, but are also used, Skarbek told me, as a sort of corporate communications system—like pneumatic tubes for prisoners.

The messages inmates send include extensive questionnaires for new arrivals. Nuestra Familia is particularly sophisticated, and, in a sure sign of bureaucratization, the gang even has an initialism for its new-arrival questionnaire: NAQ. “When you get put in your cell, and the door slams shut, you might get a fishline with a piece of paper on it,” Skarbek says. “And you’ll be expected to answer the questions in full.” The survey might include questions about your offense, your judge, and your relatives in other prisons. But it could also ask where you lived on the outside and what resources you have that could be valuable to the gang. The questionnaires are collated and checked. At some prisons, inmates use their cellphones to confirm details on Facebook, and Skarbek says they have been known to open LexisNexis accounts. Gang members are trained in micrography—the writing and decipherment of very tiny letters—so they can produce tightly rolled pieces of paper, called “kites,” to be transported from prison to prison in the usual orifice. These activity reports circulate around the prison system. Christopher Acosta showed me a kite that had been intercepted at Corcoran State Prison, reporting on a gang’s battle with a rival there.

An inmate doing push-ups in the SHU’s exercise yard, a small concrete room with an overhead skylight where inmates are allowed to spend an hour and a half a day and receive their only exposure to sunlight

Finding kites is difficult, because guards cannot cavity-search every inmate every day. The only way to control known gang members is to confine them under strict conditions that make communication almost, but not quite, impossible—no freedom of movement or circulation with the general prison population, for example, and only rare, carefully monitored visits.

Over the years, California has tried two broad strategies for gang management. The first was to break up gangs and scatter their members to distant prisons where their influence would be divided and diluted. That strategy too frequently allowed gangs to metastasize, effectively seeding the whole prison system, and even other states’ and the federal system, with gang activity. The current strategy, implemented in the 1990s, is to identify high-level gang members (a process called “validation”) and bring most of them to Pelican Bay.

Pelican Bay is far from the gangs’ strongholds of Los Angeles and the Central Valley. In every direction there is little more than redwoods, marijuana farms, and seacoast. More important, Pelican Bay has the facilities and knowledge necessary to isolate and neutralize gang members. In Sacramento, the Department of Corrections and Rehabilitation has posters on the wall showing mug shots of all the major gang leaders—the Lee Iacoccas, Steve Jobses, and Henry Fords of the underworld—grouped by the prisons they live in. Most are at Pelican Bay, probably for life, in a snowflake-shaped building called the Security Housing Unit, or SHU (pronounced “shoe”).

Of course, there are ways to control inmates that American prisons have never tried on a large scale. Skarbek points out that the gay-and-transgender unit of the Men’s Central Jail in Los Angeles County is safe and gang-free—so much so that prison officials have had to screen out straight Angelenos who play gay just to keep away from gangs. That jail is simply small and well administered, argues Sharon Dolovich, a UCLA professor who studied it, and it’s not clear that its methods could scale up. We could easily replicate less enlightened penal practices, too. “In other countries, they can use corporal punishments not available to authorities in American prisons,” Skarbek says—a bullet in the back of the neck is a strong deterrent to any Chinese gang that might form behind bars. Within the bounds of American civil rights, though, we are left with prisons whose smooth operation relies in part on the predatory activities of gangs—and with facilities like the SHU, which is California’s effort to control the gangs by subjecting their leaders to levels of surveillance and restriction far beyond what most American inmates face.

Walking into the SHU feels like entering a sacred space. After the clanging of doors behind you, a monastic silence reigns. The hallways radiate from the command center at the hub of the SHU snowflake, and each one has chambers on either side that sprout chambers of their own. The hallways echo with footsteps when you walk down them. There are no prison noises: no banging of tin cups, no screaming of the angry or insane. The silence is sepulchral, and even when you get to branches of the snowflake, where the inmates actually live, it seems as if everyone is in suspended animation, on one of those interstellar journeys that last multiple human lifetimes.

In fact, many are just watching television while wearing headphones. In the company of Christopher Acosta, I visited a cellblock where fewer than a dozen cells held men, most of them living without cell mates. Before entering, I met a female security guard who, after demanding that I display my identification card more prominently, showed me a board with inmates’ pictures on it, each color-coded. Hispanics and whites predominated. She showed me the slips of paper indicating that a couple of inmates wanted halal food, although she said she suspected the meal requests were a way to break monotony and create work for the staff, rather than as an expression of any authentic religious conviction. She said the inmates were allowed televisions with the speakers disabled, as well as 10 books at a time.

The other Pelican Bay inmates were enjoying time together in the main yards, but these hard-core gang members didn’t have that option. Instead, they could go to a large, featureless concrete room at the end of the block for daily solitary exercise. The “yard” had a plexiglass roof that allowed them to see the sky above, and a small drainage hole in the floor, through which they could sometimes communicate faintly with other inmates on other cellblocks. Last year, gang members used the drainage pipes of their in-cell toilets to communicate clandestinely across cellblocks and coordinate a hunger strike by inmates statewide, to protest the conditions in the SHU.

With a buzz and a clang, the guard opened the last door, and Acosta and I entered the cellblock. He warned me that no one would talk. We had spent much of the day discussing the violent proclivities of the men under lockdown at Pelican Bay—how they became experts at weapons craftsmanship, for example, and could fashion the metal post of a bunk bed or the edge of a cell door into a spear, known as a “bone crusher,” that could be flung from inside a cell and penetrate a man’s neck or liver. So I expected hostile interviews, if any at all.

One of the first men I saw turned out to be genial but squirrely. He was Hispanic, refused to give his name, and babbled away about how prison gangs are “just a thing,” never quite articulating what that meant. The only sentence he said that made any sense was that he was in for life for killing two people. The door that separated him from me was a steel plate with small holes in it. After just a few seconds of his talking, I got a headache, partly from his mad monologue and partly from the odd moiré effect of looking at him through the screen.

As I passed down the line of cells, I tried talking to everyone but got little response. One heavily tattooed Hispanic man flicked his hand at me from behind the steel door, as if to shoo away a flea. Most ignored me, and the few who paid any attention just stared at me like I was prey and said nothing other than “no.” Finally one man with large glasses and a thick black mustache said, “Prison gangs? There ain’t no prison gangs here.” He then turned to a blank wall and started doing calisthenics.

When I emerged, and the door had clanged again behind me, I told the guard I hadn’t managed to talk with anyone. She was not surprised. Any conversation they attempted, she said, might be overheard and used against them.

But there are limits to what even the most carefully designed prisons can constrain. The guard and I were talking in library voices, and no sounds came from the row of cells nearby. “It’s quiet,” I said, lowering my voice. “Can they hear what we’re saying?”

“Every word,” she said. “Every single word.”