China's Dystopian Tech Could Be Contagious

For all the particularities of life in China, its big cities offer a familiar cosmopolitanism. Teenagers giggle over K-Pop videos in a Chengdu Starbucks. Strollers and dog walkers compete with an almighty clutter of dockless share bikes on the sidewalks of downtown Shenzhen. In Beijing’s trendy Sanlitun district, weekend shoppers size up Uniqlo parkas like bargain hunters anywhere. Here are all the signs and markers of placeless, globalized consumerism, and the metropolitan lifestyles that go along with it.

Home to half a billion human beings, beset by pressures both shared with other urban places and uniquely Chinese, it’s remarkable that churning conurbations like Shanghai, Chengdu, or Beijing are not constantly breaking out into open, ungovernable chaos. Just like cities anywhere, though, they do not—a stability that appears to arise almost entirely from self-organization.

Perhaps spurred on by their distaste for everything implied by such liberality, the Chinese government has become convinced that a far greater degree of social control is both necessary and possible. It now has access to a set of tools for managing the complexity of contemporary life that it believes will deliver better, surer, and more reliable results than anything produced by the model of order from below.

Known by the anodyne name “social credit,” this system is designed to reach into every corner of existence both online and off. It monitors each individual’s consumer behavior, conduct on social networks, and real-world infractions like speeding tickets or quarrels with neighbors. Then it integrates them into a single, algorithmically determined “sincerity” score. Every Chinese citizen receives a literal, numeric index of their trustworthiness and virtue, and this index unlocks, well, everything. In principle, anyway, this one number will determine the opportunities citizens are offered, the freedoms they enjoy, and the privileges they are granted.

This end-to-end grid of social control is still in its prototype stages, but three things are already becoming clear: First, where it has actually been deployed, it has teeth. Second, it has profound implications for the texture of urban life. And finally, there’s nothing so distinctly Chinese about it that it couldn’t be rolled out anywhere else the right conditions obtain. The advent of social credit portends changes both dramatic and consequential for life in cities everywhere—including the one you might call home.

A dominant current of urbanist thought in the West sees order in cities as uncontrived—an emergent outcome of lower-level processes. Canny observers like Georg Simmel, Jane Jacobs, and Richard Sennett hold that virtually everything that makes big-city life what it is—and big-city people who they are—arises from the necessity of negotiating with the millions of others with whom city dwellers share their daily environments. In cities that are set up to afford this kind of interaction, people learn to practice what the sociologist Erving Goffman called “civil inattention.” They acknowledge the presence of others without making any particular claim on them. This creates the streetwise, broadly tolerant urban character of big, bustling cities from Istanbul to Berlin to Dakar, Senegal.

To hear Jacobs tell it, the city’s special heterogeneity actually keeps its communities safe, by guaranteeing that there are “eyes on the street” at all times—a critical mass of local residents, merchants, and passersby situated to observe the neighborhood’s comings and goings, and, if need be, deter the outbreak of abuse or hostility. Such organic surveillance only arises when a given locale supports a healthy mix of schedules, uses, and users, conjuring collective security out of the flux and churn of different kinds of people moving through the same space at the same time.

Admittedly, the 20th century did slowly erode the terms on which this insight was founded, as the medieval and modern incarnations of cities gave way to suburbs, sprawl, and automobiles. But Jacobs’s fundamental point still stands: Left largely to its own devices, a city forges a surprising degree of stability from its underlying diversity and complexity.

Despite its Western pedigree, there’s no reason these tenets wouldn’t roughly apply to Chinese cities and those who dwell there. Some aspects of city life, after all, remain just as true of contemporary Qingdao as they did to the 1950s Greenwich Village in which Jane Jacobs developed her ideas, despite the gulfs of time, culture, and ideology that separate them. The proprietor of a corner shop still enjoys an unusual degree of visibility into his or her neighbors’ affairs, while behavior that threatens the peace will still be policed by disapproving gossip as much as formal complaint.

As far as the ruling elites of Zhongnanhai are concerned, though, “sincerity construction,” or the process that results in stability and public rectitude, is something far too important to be left to the unplanned interactions of millions of city dwellers. From their point of view, orderliness is paramount, because orderliness makes for stability, and stability makes for continued economic growth. In their 21st-century interpretation of the “mandate of heaven,” the 3,000-year-old doctrine of Chinese imperial rule, only continued economic growth underwrites continued legitimacy.

Seen in this light, order produced from below is not reliable enough to be trusted. It leaves too much room for chance. And worst of all, from the perspective of a party bent on perpetuating its control, it does nothing to prevent the possibility of contagious urban insurrection. Social credit offers a salve to all these concerns.

In June 2014, the governing State Council of the People’s Republic of China issued a remarkable document. Dryly entitled “Planning Outline for the Construction of a Social Credit System,” it laid out the methods by which the state intended to weave together public and private records of personal behavior, and from them derive a single numeric index of good conduct. Each and every Chinese citizen would henceforth bear this index, in perpetuity.

As laid out in the proposal, the system’s stated purpose was to ride herd on corrupt government officials, penalize the manufacture of counterfeit consumer products, and prevent mislabeled or adulterated food and pharmaceuticals from reaching market. This framing connects the system to a long Confucian tradition of attempts to bolster public rectitude. The means was certainly novel, though, yoking together advanced machine-learning systems, online databases, municipal CCTV networks, and the pocket-sized sensor platforms known as smartphones. And so was the sprawling immodesty of the State Council’s actual ambition. Pushing far beyond the concerns for corruption, counterfeiting, and safety repeatedly invoked in its first few pages, the document ends up defining an order of control previous dynasties could only dream of. This truly would be a “system that covers all of society.”

The social-credit system was based explicitly on a familiar, Western model: the credit score. As a de facto reputation index, your credit score strongly conditions where you can rent, what kind of jobs or educational opportunities you’ll be eligible for, even what mode of travel you use to get around. This one number—formulated by obscure means, by largely unaccountable organizations, then used as a gating mechanism by a profusion of third parties, mostly in secret—has become what it was never meant to be: a general proxy for trustworthiness.

The Chinese genius, if you can call it that, was to take credit scoring as a tool of social discipline to its logical conclusion, building a formal public-private partnership around it. This move extends dominion across the entire range of interactions any member of modern society is more or less compelled to pursue by the very style and structure of contemporary life.

These disciplinary measures first reach individuals in the form of soft incentives, of the type familiar to Westerners from corporate loyalty programs. Citizens with higher social-credit scores enjoy discounts or upgrades on products and services, like hotel rooms or internet connectivity. Those who wear virtue on their sleeves further—perhaps by taking public transit consistently instead of driving to work, taking out the recycling regularly, or even denouncing a misbehaving neighbor—might enjoy new benefits, like being able to rent a flat with no deposit, or earning the right to send their children to exclusive schools. This hardly sounds like authoritarianism run amok, and to a certain degree, patriotic Chinese netizens are right to complain when Western critics conflate such nudges toward preferred behavior with actual tyranny.

But the system provides abundantly for sticks as well as carrots. Attend a “subversive” political meeting or religious service, for example, or frequent known haunts of vice, or do under-the-table business with an unregistered, informal enterprise, and the idea is that the network will know about it and respond by curtailing one’s privileges. The state wants its citizens to believe that there’s little point in trying to evade detection of such acts, especially when they are strongly correlated with suspicious sites, either by mobile-phone location data or by China’s extensive national network of facial-recognition-equipped surveillance cameras. Once detected, the system promises to pass judgment on the things a citizen is and is not permitted to do, buy, or access. And with no recourse in real time, no ability to appeal, and nowhere to turn for help.

The consequences are nontrivial. The scheme is fused to a repertory of existing techniques the Chinese state retains to guarantee control, including the “hukou” system of internal passports that restricts the free movement of individuals, right up to detention, incarceration, and execution. What emerges is a marriage of database and truncheon—a vision of supple, gleaming technology at the surface of everyday life, working hand in hand with the oldest and most brutal forms of oppression, continuing their unbroken reign in the depths below.

That is the vision, anyway. What does social credit look like in practice?

The Chinese state relies upon private enterprise to implement social credit and extend its tentacular reach. Top-tier concerns like Alibaba’s Alipay, Tencent, Baidu, the mobility-on-demand service Didi Chuxing, and the massively popular dating site Baihe either contribute significant elements of the system’s architecture or incorporate its rulings into their services. As is common in Western-style credit scoring, these companies use readily observable and easily measurable acts as proxies for behaviors that are more abstract or harder to gauge. For example, a propensity to spend hours playing online games becomes an index of indiscipline in and of itself, even if the person playing them is debt-free and has a clean record otherwise. One’s brand of smartphone serves as a stand-in for social class, while routine purchases of diapers are taken as an indication that one is a parent and therefore more likely to be responsible. All of these qualities are factors in determining what score one is ultimately awarded.

China learned to bind real-world consequences with such clunky, even laughable inferences from the West, where algorithmic credit-scoring systems can flag a tendency to fill out loan application forms in capital letters exclusively as a proxy for low reliability and creditworthiness. But by drawing conclusions about citizens’s political sentiments into the judgments it renders, social credit goes far beyond anything currently imagined in the Western world. The state relies on its corporate partners to punish behavior it regards as problematic. Private, commercial transactions—renting a car, reserving a hotel room, buying a plane ticket—become venues for state-directed punishment for nonconformity.

It’s not always clear, even in retrospect, what behavior might have triggered such an exclusion, but early indications suggest that a broad array of putative transgressions can be picked up and punished by the system. Human Rights Watch reports that the lawyer Li Xiaolin was barred from boarding a domestic flight because a written apology he’d offered a regional court in an unrelated matter some months before was held to be “insincere.” His name had been entered into a national no-fly list because he’d run afoul of the whims of a petty bureaucrat. And the activist Liu Hu was apparently ensnared by the system for undertaking political activity abrasive to the state. Frozen out by the network, he can no longer buy real estate, secure a commercial loan, or pay for airfare. He can’t even travel on the national high-speed rail network. It’s not hard to see how such restrictions, applied broadly enough, would put an effective brake on nonconforming behaviors—or even the expression of nonconforming opinions.

The implementation of social credit has been piecemeal so far: partial, local, and experimental. And there have been some encouraging signs of popular resistance to imposed “sincerity.” In the coastal province of Zhejiang, The Wall Street Journal’s Jeremy Page and Eva Dou report that locals have rejected a digital tool intended to let them inform on neighbors, on the grounds that it reminds them of the Cultural Revolution–style denunciations that remain within the living memory of elders.

Nevertheless, today’s ruling party has had many years to refine the art of what longtime China observer Rebecca MacKinnon calls “networked authoritarianism.” The components for a comprehensive, end-to-end inscription of sincerity culture are readily available. China is an unabashedly authoritarian state equipped with everything it needs to cut dissent off at its source. It has the power to create a generation of compliant subjects both unaware of alternatives and utterly unable to formulate whatever grievances they might hold in a politically potent way.

This descending shroud has implications that extend to suburban and rural environments as much as urban ones. But a robust social-credit scheme has special impact on city life. If urbanists like Simmel, Jacobs, Sennett, and Goffman concluded that citizens’ ability to get along with one another in densely settled places is an emergent matter, social credit suggests just the opposite: Urban order, from outward harmony to inner sincerity, can (and should) be imposed from above.

There’s something about this notion that’s inimical to the give-and-take of big-city life. The fear of running afoul of arbitrary penalties suppresses the churn and friction central to urban experience—especially once that fear becomes an unconscious driver of personal choices. When the Alibaba executive vice chairman Joe Tsai urges people to understand the links between online behavior and real-world sanction “so they know to behave themselves better,” urbanists worldwide should take heed. What Tsai means is that Chinese society—and by extension, Chinese cities—will run more smoothly once citizens have internalized the idea of an ever-watchful state and learned to comport themselves accordingly. It’s hard to imagine a more concise illustration of social credit’s capacity to flatten and dull the ferment cities rely upon to create vitality, meaning, and value.

It’s no longer surprising that inventions born in Silicon Valley wind up being adopted around the world, disrupting long-standing social conventions and arrangements. So why shouldn’t the same thing be true when innovation appears with Chinese characteristics?

As private enterprise takes an increasingly prominent role in the creation and management of ostensibly public urban space, as neo-authoritarianism spreads unchecked, and as pervasive technology weaves itself ever more intimately into all the sites and relations of contemporary life, all of the material conditions are right for Chinese-style social credit to spread on other ground. Consider what Sidewalk Labs’ neighborhood-scale intervention in Toronto implies—or the start-up Citymapper’s experiments with privatized mass transit in London, or even Tinder’s control over access to the pool of potential romantic partners in cities around the world—and it’s easy to imagine a network of commercial partners commanding all the choke points of urban life. The freedoms that were once figured as a matter of “the right to the city” would become contingent on algorithmically determined certification of good conduct.

It would be remarkably easy to achieve. It’s just a matter of making explicit the determinations that already go into credit scores—of binding together the data brokerages that even now siphon up public records, social-media profiles, web searches, and similar digital traces of life here in the West, and making our rights and privileges as city dwellers and citizens contingent on what they infer from our behavior. Unless this tendency is contested and defeated now, what has unfolded in China since 2014 might become an early preview of the way order is achieved and maintained in the cities of the 21st century.