Why New York Subway Lines Are Missing Countdown Clocks

“I honestly just wanted to know why the F train didn’t have clocks. I never expected it to be so complicated.”

There are people who stand every morning outside the Carroll Street station in Brooklyn staring dead-eyed into the middle distance. They stand still in ones and twos, clearly strangers to one another, mostly quiet, as though they’d stopped on their way to work to take note of some spectacular disaster in the sky. But you look in the general direction they’re all looking and there’s nothing there.

They are, as it turns out, waiting for the F train. Carroll Street is one of the rare New York subway stations whose trains are boarded underground but where you can stand outside to see them coming. When you spot the F rolling down the bridge, you have just enough time to run inside to catch it. So people stand there waiting. They wait for as long as it takes, for as long as their patience will allow, because in 2015 there is no app, no screen, not even a scratchy voice over a PA system that can tell them when the train is actually going to arrive.

But here’s the truly crazy thing: The only people who know exactly where that train is are on the train itself. The signal-tower operators don’t know; there’s no one in the Rail Control Center who could tell you, because the F isn’t hooked up to the Rail Control Center. Today, for the F train—along with the G, the A, B, C, D, E, J, M, N, Q, R, and Z—the best the system can say is that the train will get there when it gets there.

This is both infuriating to riders who want to be able to plan their commutes—spend those extra ten minutes at home, or forego the subway altogether if there’s a long delay—and a symbol of a broader failure. It’s hard to say what, exactly, but something important seems to have gone wrong when the tracking apparatus for subway cars is worse than it is for pizzas.

The best estimates today are that countdown clocks that tell you when the next train is coming will arrive on the so-called B division of the New York subway system in 2020. (The A division already has them.) That would make them about nine years overdue. It is easy to take for granted that governments move slowly, particularly on large infrastructure projects, particularly when those projects involve software. But we live in a world with cars that can drive themselves. Trains are huge objects that move in one dimension. How could it cost hundreds of millions of dollars and take nearly a decade just to figure out where they are and report that information to the public? Really: How?

* * *

There is a locked door just below the southwest entrance to the 14th Street A/C/E station. It looks like the door to a small office or break room. But going inside is like stumbling into the Keebler tree. There is a whole world down there: a warren of rooms and equipment, including a working section of track about 20 yards long, that comprise the subway system’s Signal School. This is where the MTA teaches its staff to operate the subway’s switches and signals, which look like simple traffic lights but turn out to be key components of one of the earliest and most complex manmade information-processing systems in existence.

I’m there one morning in early October, one stop on the road to answer what I thought would be a simple question: Why don’t we know where all the trains are?

I’m watching Wynton Habersham, the vice president and chief officer of Service Delivery, play with a model train. It’s about a foot long. It looks like a toy next to the full-sized track. It basically is a toy, except that it’s hooked up to real signals. Habersham is using it to explain how they work.

The basic idea is that you don’t want trains to run into each other. So you put signals along the track that tell drivers to slow down or stop when the track ahead of them is occupied.

All track on the New York subway (and on most American rail) is broken into sections, here about 1,000 feet long. An electric current is constantly running in a loop through each section. When a train enters a section, it short-circuits the loop, which allows the system to know that the section is occupied. The signals behind it automatically turn red.

Fixed-block signaling systems, in use since the late 1800s, keep trains from getting too close to one another. (Regional Plan Association)

The neat thing about subway signals, as opposed to the ones you find on the road, is that they actually force you to stop. When a signal is red, a footlong metal T called (appropriately) a “train stop” protrudes above the track; each train car has a corresponding “trip cock” on its wheel frame connected to the emergency brakes. If you were to drive by a stop signal the train stop would hit the trip cock and you’d screech to a halt.

That’s good, but only if you can stop in time. And indeed, the signals are set up so that if you have to force a fully-loaded, full-speed train to stop, there’s always enough space before it collides with anything else. In principle this is easy to do: When a section is occupied, don’t just make the signal behind it red—go back as far as the maximum stopping distance and make all those signals red, too. As trains run through the system, they’ll leave a wake of buffer space behind them, a trail of red and yellow signals.

When New York’s signaling system was first being installed—much of the work took place at the turn of the last century, with another big wave during the FDR-fueled construction boom of the 1930s—the designers wanted to make it impossible for there to be a collision. They wanted intelligent signals: signals where even if you made a mistake, even if you wanted to, you couldn’t command two trains to simultaneously drive into the same section of track.

This becomes especially important when you have tracks that cross, and switches that route trains from one track to another. In a subway system this happens all the time. New York’s is especially hairy, since many lines have both an express and local track going in each direction. Some big stations can have as many as a dozen lines connecting.

The way you do it is by having what’s called an “interlocking.” An interlocking is just a configuration of signals, switches, and their controls that is set up in such a way that you can never have an unsafe state. In the early days, this was enforced by levers that literally interlocked. If you want to flip a switch over here, you first have to make the signals over there red. If a train entered this section over there, the switch over here would always be disabled.

Bernard Greenberg, who did graduate work in Computer Science at the Massachusetts Institute of Technology in the early ‘70s and on a whim created a fully functional computer simulation of most of New York’s interlockings, explains that “railroad interlockings and telephone exchanges were the big early computers.” Before semiconductors and the modern microprocessor, these systems represented our best attempt to mechanize complexity.

To design even a single interlocking was, and still is, “terrifically complicated.” A thousand considerations—the interactions between signals, switches, and trains; the curves, grades, and other track conditions that affect train speed and braking distances—go into their design. Greenberg, also a composer, likens the problem to writing classical music. “Interlocking is a kind of counterpoint for subway tracks; each track adds about as much complexity as a line of a fugue.”

A 1968 plan for a small section of New York’s trackage and interlocking, showing the placement of signals and the part of the track whose safety each signal controls. (nycsubway.org)

The switches are connected by wire or still, in some cases, by actual levers, to rooms called “towers” where operators can see which sections of track are occupied, what color the signals are, and what the state of each switch is. By pulling levers and talking to drivers over the radio, they clear paths for trains. The interlockings ensure that even if they do something wrong, they can never clear two trains to take the same path.

A view of the interlocking panel at the signal tower in Bowling Green station (Jersey Mike's Rail Adventures)

The reason there are no realtime countdown clocks on the F line is that even the tower operators don’t know which train is where. All they can see is that a certain section is occupied by a certain anonymous hunk of steel. It’s anonymous because no one has a view of the whole system. A hunk comes into one section of track from somewhere else; the tower’s job is to get it through their section efficiently. The next tower they pass it to will likewise not know whether it’s an F, say, or a G. When there are incidents, trains are located by deduction.

This complex—of towers, signals, switches, and track sections—is responsible for a disproportionate share of the costs and foibles in the operation and maintenance of New York’s subway system.

The equipment is old and breaks all the time. In fact it’s so old that the MTA can no longer buy replacement parts from the manufacturer; it has to refurbish them itself. Some of the controls for the interlockings are originals from the 30s. Much of the wiring is still insulated with cloth, instead of rubber; ten years ago the entire Chambers Street interlocking caught fire. Salt water from Hurricane Sandy did damage to trackside switches and signals that is still being repaired.

Inside the Signal School there is equipment from every major era, since it’s all still active in various parts of the system. As a demo, Habersham at one point flips an old-style switch on the big replica track. It lets out a giant pneumatic wheeze, as though the tired station itself were sighing. Even the little toy train that he used to demonstrate signaling basics is falling apart; there was so much rust and dust on the tracks that at several points another MTA employee had to help it along with his hand.

In all there are more than 12,000 signals and 300,000 relays that comprise the system’s interlockings. Signal failure is the largest source of delays in the subway. There is an incident on average every 11 hours. Whenever a signal fails, the ones behind it go red, since they can’t be sure there isn’t a train in that section. Often, a can or a piece of aluminum foil is enough to fool a track circuit. “Something as simple as that can create a Conga line of trains,” Habersham says.

Habersham and the whole of the MTA are unexpectedly forthcoming about the sorry state of the signaling system. They even put out a video that gleefully shows off the worst stuff. Look at how broken it is! Look at how old!

Their point is not to flagellate themselves. It’s to drum up capital support for a systemwide upgrade and in particular for a program called CBTC. The MTA, too, thinks it’s ridiculous that all a tower operator knows when they look at their board is that some hunk of steel—which one, they can’t be sure—is sitting on some section of track. You want fewer delays? You want realtime countdown clocks?

CBTC is the answer.

* * *

For decades, New York had little interest in modern signaling technology. It was happy just to keep the trains running. The MTA was biased in favor of brick-and-mortar work. The attitude was that if you weren’t building a new station or track, it probably didn’t need to be done.

Then they got a wake-up call. Just after midnight on August 28, 1991, a 4 train running downtown with more than 200 passengers derailed at a switch just short of Union Square. Five people died. So many support columns were destroyed that the street above the station sunk by half an inch. The driver had been drinking heavily the day before the accident and was asleep as the train approached the station. They were traveling at 50 miles per hour, five times the normal speed.

A train stop on the tracks triggered the emergency brakes but it turned out to be too little too late. The signals had been spaced too closely; there wasn’t enough track to slow the train. It was a sobering moment. The foundations of the system—track circuits, signals, train stops—were cast into doubt.

By 1994 the MTA was presenting a business case for a new kind of signaling system that had become the standard for modern transit projects: communications-based train control, or CBTC.

CBTC does away with the “fixed-block” signaling system, where track is broken into sections that report whether they’re occupied. Instead, each train is equipped with a radio and onboard computer that identifies its precise location, and coordinates that information in real time with a central control center and other trains to decide exactly how fast it can safely go. Trains therefore run with a moving window around them, which constantly shifts depending on their own speed, size, track conditions, and traffic.

The big benefit is that this allows you to run trains closer together. And you can use the track more flexibly. For example, in many fixed-block signaling systems, including New York’s, the signals along certain sections are only set up for trains to go in one direction. But CBTC just sees a bunch of track; it can automatically figure out which parts of it are available, say, for turning trains around.

A diagram of a moving-block signaling system powered by CBTC (Regional Plan Association)

In fact, with CBTC, you no longer even need signals or interlockings. If you were willing to go all-in, you could completely remove all your old levers, relays, track circuits, and signal towers. Safety would instead be guaranteed by the fact that the system as a whole knows about every train and can control its speed. If there were ever a communications failure, failsafe equipment on each train could instantly trigger the emergency brakes.

The only equipment required are trackside controllers—computers with radios, basically—spaced at regular intervals; train controllers installed on each train; and a central control center. Unlike the old relays, all the trackside equipment is completely enclosed and can withstand decades of wear and even saltwater.

Maintenance costs would plummet. The bulk of the system’s complexity is moved into software: software that interfaces with the physical train controls; software that understands the entire system’s track layout, including all switches, grades, and stations; and software with a huge number of rules about what kind of movements are allowed when.

Since CBTC requires realtime location data to operate, relatively little additional work is required to give that data to customers, say in the form of a smartphone app that reports arrival times or in countdown clocks hung at stations.

The Canarsie Line running from 8th Avenue in Manhattan to the Canarsie neighborhood in southeast Brooklyn was chosen to pilot the technology. It was one of the oldest lines in New York, starting life as a steam railroad in the early 1900s, but seemed ideal as a test case: It was completely isolated from the rest of the system, with just one line, the L, running along it; it was shorter than most other lines; and it had relatively low rider volume.

Work began in 1999. It wouldn’t be fully operational until 2011. At the current pace of installation, the subway system as a whole won’t be converted to CBTC for another 175 years. It will cost $20 billion.

* * *

The Regional Plan Association has its offices at 4 Irving Place, in the Con Edison building. I went there one recent morning to talk to Richard Barone, the director of the RPA’s Transportation Programs and the author of their 2014 report on CBTC, “Moving Forward: Accelerating the Transition to Communications-Based Train Control for New York City’s Subways.” The RPA is a kind of urban-policy think-tank that focuses on the tri-state area. Its main outputs are tomes it calls Regional Plans, the first three of which were released in 1929, 1968, and 1996. The fourth is due to come out soon.

Barone and I walked through a room full of maps to a nondescript corner office on the 7th floor. On the bookshelf was a copy of The Power Broker. In graduate school, Barone told me, he studied labor relations.

I was there to find out why CBTC was taking so long and costing so much.

“You try to benchmark New York to other places and you can’t,” he said. Everything is harder here. Everything takes longer here.

He explained that the Canarsie pilot suffered from problems that weren’t unusual for big transit projects in New York.

The first was outmoded work rules. CBTC is designed so that trains can run themselves. But the L still has two-person crews on board every train. They’re not very busy: An April 2007 article titled “Look, Ma—no hands!” in the trade magazine Railway Age featured a delighted train supervisor named Lance Parrish riding in a CBTC-equipped train on the Canarsie Line. “All Parrish has to do is scan the onboard displays and acknowledge a flashing/beeping alerter every 20 seconds.”

The second was a fear of change. It costs $168,000 per track-mile per year to maintain trackside signals, 90 percent of which is spent on labor—much of it done overnight and on weekends, qualifying the workers for overtime. If those signals were eliminated, millions of dollars could be saved each year. But New York decided to run CBTC on top of a reduced form of the old fixed-block signaling system, requiring that both be expensively maintained, despite evidence from other cities that no backup was necessary. (In Vancouver, the SkyTrain has had no CBTC-related accidents in more than 26 years.) And the fact that the two systems had to work together—requiring the supplier to study the old signals in depth—became a major source of delays.

Barone says New York just wasn’t willing to rip the band-aid off. Cities like London deal with major transit upgrades by packing maintenance and line closures into as short a window as possible, however painful that might seem at the time. New York, by contrast, draws out its track maintenance. When I spoke to the president of Thales Transport & Security, one of two major CBTC suppliers to New York, he said that “getting time on the track is by far the biggest schedule driver.” Crucial test-runs get queued behind miscellaneous track maintenance, so that it takes months to validate even small changes. “In the New York mindset,” he said, “there just isn’t the concept of the trains ever stopping.”

All that waiting isn’t free. These are huge projects for a company like Thales; they’ll spin up a whole office, a whole mini workforce, just to work on it. And when they’re waiting for track time, that workforce doesn’t just spin down—it continues to get paid. Anticipating delays, contractors inflate their bids.

All told, CBTC enabled a 3 percent decrease in end-to-end travel time on the L, and the MTA claims that it has increased capacity from 12 to 15 trains per hour in the late ‘90s to 20 trains per hour today. (It’s not clear that CBTC is responsible for this change. In 1999, the Williamsburg Bridge was closed temporarily, and the MTA managed to increase service on the L to 20 trains per hour to make up for lost capacity on the J, M, and Z lines.)

The project took 11 years to become fully operational and cost $340 million. The next steps for CBTC are the Flushing line, which is already underway, and the Queens Boulevard line, where the contract was just awarded. It is shaping up to be a more capital-intensive and longer-running project than the first leg of the Second Avenue subway, and almost as big as Boston’s Big Dig.

But compared to those projects it is awfully nebulous. It’s hard to explain an underground-systems upgrade to customers; and it’s hard for customers to notice, especially if the city keeps maintaining the old signal systems and neutralizing its own cost savings.

That’s why the MTA has tried to associate CBTC with countdown clocks. New York riders crave realtime information about trains. They don’t care how they get it. So when Transit wants to drum up support for an obscure, costly, many-decades-long capital project to upgrade to CBTC, they always point to the clocks. (“Sustained Investment Makes Real-Time Information Possible,” declares one 2012 press release.) Reporters, struggling to make sense of a half-dozen interrelated projects, follow the MTA’s lead and assume that realtime train-location information depends on signal upgrades.

But that would make for some pretty expensive clocks, and it would make them awfully long in arriving. The F train, for instance, if it had to wait for CBTC to get realtime arrival information, wouldn’t see it until 2035.

It’s a misleading narrative. You can get countdown clocks without touching the signals. The MTA knows this. They learned the hard way that tying countdown clocks to a massive CBTC-esque signal upgrade is a mistake. That’s what happened with the 1-6 lines: The project, called ATS, cost $220 million and took 5 years longer than it was supposed to. To deliver countdown clocks to the rest of the lines by 2017, they intentionally developed a simpler plan. The whole point of it was to avoid trackside signal upgrades.

If they’re being realistic, it’s that project, not CBTC, that the MTA should be talking about when it talks about clocks. The thing is, it’s kind of awkward to talk about. That project is to ATS as ATS was to CBTC: a lesser, overlapping, incompatible and possibly unnecessary stopgap. One of three redundant initiatives started at different times for different reasons, by an agency clumsily trying to rewire itself.

The story of how it could take 28 years to install clocks that tell you when the damn trains are coming turns out not to be about some dinosaur fixed-block signaling system and the gleaming new technology here to replace it. It’s simpler than that: It’s the story of a large organization’s first encounter with a large software project.

* * *

New York’s subways were initially built by two different companies, the IRT and the BMT, with different-sized cars and tunnels. Today there are still two systems that run more or less independently, in the sense that trains from one can’t run on tracks from another. These form the A division (the numbered lines) and B division (the lettered lines) respectively.

In the mid-90s, the MTA decided to make major upgrades to the older of these two systems, the A division, devising a $200 million plan to repair old signals, convert mechanical interlockings to use electronic relays, digitize the interlocking data, pipe that data into a central control center, use it to activate switches remotely and reroute trains, and distribute arrival information in real time to countdown clocks installed in every station on the 1-6. It would be a massive consolidation of what had been disparate systems. It was going to require serious technical expertise: not just to install the new gear, but to retrofit and interface with the existing interlockings. The project was called “ATS,” for Automatic Train Supervision.

If it sounds similar to CBTC, that’s because in all respects the goals are the same, except that ATS stops short of allowing remote control of trains. Both approaches consolidate location data that had been distributed and use it to build a picture of the system as a whole, allowing operators to put a name to each train and therefore to make better decisions about how to route them. But CBTC, when implemented as designed, is by far the simpler way to do this, since it obviates signal towers and interlockings altogether.

Given that CBTC would—and will, someday—accomplish the same thing as ATS; and given that it could be done much more efficiently; it makes you wonder how ATS got started in the first place. The answer is that planning for ATS began in 1992, two years before the MTA started seriously considering CBTC as an option. Had ATS been started two years later, it might have never been started at all.

The project was in many ways a success: It brought old signals into a state of good repair, it created a (very cool-looking) centralized Rail Control Center, and it delivered countdown clocks to the 1-6 lines. But all of that was supposed to take nine years; it took 14 instead. ATS is now remembered as a classic case of mismanagement.

A post-mortem by the Federal Highway Administration details how from the start, an agency which had had little experience with large “systems” projects tried to wing it. For instance, the consulting firm tasked with developing the project plan never made a list of requirements, didn’t talk to the workers who would be maintaining the system until after it was designed, and left vague instructions for large chunks of work—specifying, for instance, “similar functionality to what is currently available”—that later became the focus of drawn-out contract disputes.

The MTA thought that they could buy a software solution more or less off the shelf, when in fact the city’s vast signaling system demanded careful dissection and reams of custom code. But the two sides didn’t work together. The MTA thought the contractor should have the technical expertise to figure it out on their own. They didn’t. The contractor’s signal engineer gave their software developers a one-size-fits-all description of New York’s interlockings, and the software they wrote on the basis of that description—lacking, as it did, essential details about each interlocking—didn’t work.

Gaffes like this weren’t caught early in part because the MTA “remained unconvinced of the usefulness of what seemed to them an endless review process in the early requirements and design stages. They had the perception that this activity was holding up their job.” They avoided visiting the contractor’s office, which, to make things worse, was overseas. In all, they made one trip. “MTA did not feel it was necessary to closely monitor and audit the contractor’s software-development progress.”

The list goes on: Software prototypes were reviewed exclusively in PowerPoint, leading to interfaces that were hard to use. Instead of bringing on outside experts to oversee construction, the MTA tried to use its own people, who didn’t know how to work with the new equipment. Testing schedules kept falling apart, causing delays. The training documentation provided by the contractor was so vague as to be unusable.

You get the impression that the two groups simply didn’t respect each other. Instead of collaborating, they lobbed work over a wall. The hope on each side, one gathers, was that the other side would figure it out.

Miraculously, they finished. It took nearly twice as long as they thought it would. After everything, after more than a decade, they were left with a system that isn’t even compatible with CBTC. And they were still less than halfway done.

* * *

On the B division, all train activity is still written down on paper and sent in batches to a central location for processing.

Plans to bring ATS to the lettered lines miscarried more than three times, in 2006, 2008, and 2010. There just wasn’t the will to go through with it again, let alone on a division nearly twice the size of the A.

But once the public had countdown clocks on some lines, they couldn’t understand why they didn’t have them on the others. “An interesting dynamic developed in response to the ATS-A system’s ability to make predictive train arrival announcements,” writes the MTA’s Thomas Calandrella in a report. “A customer-driven campaign, filtered through city and state governmental representatives, demanded that the train-arrival signs on the 1 through 6 lines be made available to all NYCT customers. Based upon this pressure, the agency committed to providing predictive train-arrival information to the customers on the platform and on the web by 2017.”

They hope to accomplish this with a new project called ISIM-B. It stands for “Integrated Service Information and Management.” It is meant to be like the better parts of ATS reborn, this time leaner, simpler—not so much an omnibus system as a collection of smaller ones.

Recall that ATS was just for the A division. (That’s why it’s sometimes called “ATS-A.”) Separately, CBTC—which is like ATS+, since it includes the ability to remotely control trains—was started on a few individual lines. ISIM-B was conceived to finish where ATS had left off.

ISIM-B gained traction inside the agency when a 2012 pilot program proved that if your only goal was to get train arrival information, you could cheaply digitize the signal data from a single line and feed it into the central system.

Instead of rebuilding trackside signals, most of the upgrades on ISIM-B will take place in signal towers, meaning trains won’t have to be taken out of service. All that has to happen is for the so-called “indications” from the track circuits—the data about which hunk of steel is on which section of track—to be digitized and centralized. Only when the indications from all the towers are combined can you be sure that that hunk of steel is that F train. (Once you have a view of the whole system, instead of just a part of it, you can trace a single hunk as it moves from section to section—giving it a persistent name.)

The focus is on delivering incremental value, rather than doing any kind of total overhaul. On one view it represents a retrenchment after the drawn-out battle that was ATS. On another, it’s what ATS was meant to be in the first place. Either way it sounds like an improvement, except that according to the last two MTA capital plans, the cost for ISIM-B is projected to be in the hundreds of millions of dollars and counting. Kicked off in 2011, it was expected to be finished by 2017, but already the date is slipping: The Wall Street Journal reported a few weeks ago that funding for the project had been cut from the latest capital plan, pushing the clocks’ arrival to at least 2020.

The problem is that the project has slowly taken on a bigger and bigger scope. The minutes of a 2012 Capital Program Oversight Committee meeting reveal that initially, the project’s focus “was to provide Train Arrival Information in stations.” Several service incidents, including a winter storm, drove the MTA to “re-focus project priority to provide centralized service-monitoring and information... followed closely by customer information.”

It is growing to look more and more like ATS. A request for proposal as recent as six months ago—back when funding looked more secure—called for a 77-month software contract to build out a sophisticated Rail Traffic Management System as part of ISIM-B. That piece of the project is envisioned as a complex centralized “expert system” that would allow operators to quickly diagnose service problems and would intelligently suggest ways to work around the disruption. It is, in a word, ambitious. And ambition is the death knell for big software projects. It’s what made ATS such a quagmire in the first place. It is, one suspects, why funding for countdown clocks has been cut from the latest capital plan: The rest of ISIM-B costs too much. It costs too much because it is trying to do too much. The consequence being that for five or six years, customers will hardly see anything get done at all.

The strange thing is that the MTA has it in them to do things differently. Really differently.

* * *

In the late ‘90s, just as ATS and CBTC were promising to rebuild the informational guts of the subway, a project was cooking aboveground to bring arrival-time information to buses.

The earliest attempts were in the usual style: A single big contractor was given a fixed-fee award to do the whole thing. In 1996 it was Orbital Sciences Corp. who got the contract; they were fired four years later after making little progress. In 2005 Siemens won the bid for a $13 million pilot program to put clocks in the stations on six bus routes; that project was canceled a year later. (Siemens was the same company who worked on ATS. A 2009 report by the MTA's oversight organization shows that despite the contractor’s poor performance on that project, ATS program managers had been told by their supervisor not to give Siemens an overall “Unsatisfactory” rating, for fear that this would preclude it from participating in future projects.)

In the spring of 2010, under the leadership of chairman Jay Walder, who was known both for straining labor relations and for having an entrepreneurial streak, the MTA went in a different direction. They hired a small team of software-savvy MIT grads to come in-house and manage the bus project. Instead of procuring a single contractor, they defined the specifications for the project themselves, broke it into pieces, and brought contractors on to build each one.

Their big idea was actually to do less. Walder felt that the MTA’s credibility was at stake. He knew riders wanted realtime arrival information. He worried that failing to deliver something tangible in the first half the new administration might threaten the next capital plan. So he decided that BusTime, as the project became known, would deliver the minimum viable product. Where previous efforts had tried to install countdown clocks at every stop, this time around they would make a website and app that delivered location data directly to riders. It wouldn’t be ideal—lots of riders, particularly lower-income New Yorkers and the elderly, didn’t have ready access to the web—but it eliminated a huge part of the project’s scope.

Everything was done on the cheap. They paid their contractors competitively but required them to write open-source software, which in the long run meant the MTA wouldn’t have to pay licensing fees. For hardware, they used commercial-grade GPS units.

Having full-time software experts running the show turned out to be crucial. Previous incarnations of the project didn’t have a technical leader at the MTA—just old-school senior managers who would try to wrangle the contractors by force of will. The new in-house team, by contrast, was qualified to define exactly what they wanted from software providers in terms those providers could understand. They were qualified to evaluate progress. They could sniff out problems early.

A working version was available on a route in Brooklyn by February 2011, just months after the project began. It came to Staten Island by January 2012, and expanded to all of Manhattan by late 2013. All-in it cost on the order of $10,000 per bus.

Walder now works in the private sector. BusTime turned out not to be the new normal for the MTA—just the exception that proves the rule. In August of this year, for instance, the MTA began the next phase of its CBTC rollout with a $156 million contract. It was awarded to Siemens.

* * *

Barone—the author of that CBTC report—and I are sitting at a conference table out in the RPA common area. (We’d been squatting in the corner office.) We’ve been talking for about an hour. All the “ATSes” and “CBTCs” and “ISIM-Bs” are starting to take their toll.

“One of the things that was most frustrating when doing this work,” Barone says to me, referring to preparing the report, “was the murkiness. And the lack of uniformity in how each of these systems is being done.”

“It seems to me that there are concurrent projects going on that—” He trails off, thinking for a minute. “It’s like, you’re building ISIM to find out where the trains are located—but CBTC does that. You’re spending money to get your interlockings to be centrally automated, yet CBTC can do that too… ATS initially came out before they really thought about moving to CBTC, and therefore the first ATS is not even compatible with it. It can’t plug in. There’s a whole plan now to do a new version of it…”

He seemed weary. I certainly was. I told him I honestly just wanted to know why the F train didn’t have clocks. I never expected it would be so complicated.

The MTA has a thankless and extremely difficult job: They have to keep the trains running. They have to do it with equipment from the 1930s, in a hostile funding environment, as administrations come and go, as public interest comes and goes, in the face of storms and accidents and pieces of aluminum foil. This they manage to do. 1.6 billion people every year take the New York subway. The system carries more than 60 percent of all people coming into Manhattan every day. It is, for the most part, safe, affordable, and there.

But still, a reasonable person looking at three projects that aim to do roughly the same thing, projects that have different teams and different agendas and seem to take, always, five years longer than planned and seem to cost, always, hundreds of millions of dollars—well, this person has to wonder whether there’s some law of the universe that makes large government software projects an expensive drag or whether in fact there’s a better way.

I keep thinking of Healthcare.gov. Everyone knows that the initial project was a costly disaster, but less well known is that a small team came along and saved it. The story includes this remarkable fact: The old system cost $250 million to build and $70 million a year to maintain. The new system—which actually worked—cost about $4 million to build; its yearly maintenance was about $1 million.

If the initial system hadn’t been such a public failure, we might not have learned that it was two orders of magnitude more expensive than it needed to be. How many more projects are being run at 60 times their actual cost?

Should we stand around waiting to find out?

Related Video

Damon C. Scott is a well-known subway performer, but he's also the voice of the #1 hit song, “Look Right Through.”