Inside the Failure of

On a project this big, there were a million things that could go wrong. And they all did. AP Photo/J. Scott Applewhite

There was a lot riding on

The site has to do many things: verify a person’s identity, legal residence, and income; record his personal information; match him with health insurance plans for which he is eligible; calculate whether he’s eligible for a subsidy (and how much); help him compare plans by costs and features; and, finally, enroll him in a plan.

Enrolling in any insurance plan can be tricky; comparison-shopping for one in a state that may offer hundreds is a daunting task. It requires working with all 50 states (the exchange may be federal, but the insurers aren’t) and dozens of insurers.

What’s more, since the Affordable Care Act was designed to support state exchanges, parts of the site’s spine – specifically the data hub that pings the IRS, Social Security Administration, the Department of Homeland Security and credit bureaus – has to work with each state’s websites, federal departments, and credit agencies. That’s a staggering data and software challenge.

A lot could go wrong and did. Here’s the story of how it happened.

Like every modern website that has to make something happen rather than just display text, separated its site into a front-end that interacts with the customer and a back-end that (among other things) fetches and matches data and moves your information through the system. But unlike most enormous modern websites that make things happen – Google, Facebook, Amazon – divided this work among a variety of companies. For obvious reasons, there is no company with experience building a federal health insurance exchange.

The most important of the companies running the back-end are CGI Federal, which built the insurance marketplaces, as it had for state Medicaid programs, and QSSI, which built the data hub to verify citizens’ eligibility. The homepage is by Development Seed, a small, young D.C.-based company that came in as a subcontractor under Aquilent, which navigated the bureaucracy and bid for the contract. “If I were to bid on the whole project,” Development Seed’s Eric Gundersen told Slate’s David Auerbach, “I would need more lawyers and more proposal writers than actual engineers to build the project. Why would I make a company like that?”

The entire project was overseen by the Centers for Medicare and Medicaid Services (CMS), part of the Department of Health and Human Services. CMS wasn’t a software company, and had no experience running a project this large, but some government agency had to be in charge. CMS was also busy monitoring and coordinating the development of all the state exchanges.

Keep in mind that when the Affordable Care Act was passed and the government contracts were doled out, was a relatively tiny piece of the overall health exchange pie. The real plums were the state exchanges: There were more of them, and most of them paid more money. CGI’s contracts to run exchanges in Colorado, Massachusetts, Vermont, and Hawaii were worth more than twice as much as its contract for ($71 million); New York’s exchange was worth $183.6 million for Computer Sciences Corp., and California’s $359 million for Accenture. wasn’t supposed to be the primary end-to-end portal for citizens seeking health care.

It was supposed to be no big deal.

To understand how misfired, look at how it came about. The site is fundamentally a compromise between two increasingly polarized groups: government bureaucrats and software developers. In an attempt at moderation and to foster unity, the Obama administration tried to bring those two groups together, blending ideas from both sides, even as experienced bystanders warned that the two were incompatible. The result is a complicated mess, only capable of lurching from crisis to crisis. The only thing worse than the reality is the rhetoric, as both sides point fingers.

The fundamental technological conflict is between how software is now developed and how traditional government IT works. Most web applications built today in Silicon Valley’s orbit are agile, open, iterative. You test a minimum viable version of your application and refine your approach as you add more features and users. You move fast, break things, and stitch them back together again.

The federal government doesn’t work that way. It is built to accommodate and coordinate monoliths and their schedulers. And so it gravitated toward IT companies that speak that language: IBM, Deloitte, CGI Federal, Aquilent. Those are companies with enormous legal teams that architect and plan their projects well in advance to deliver full products on budget and on schedule. (This hardly ever happens, but let’s indulge the fiction.)

That approach isn’t much good for delivering state-of-the-art services over the web, so neither the agile developers nor the old government IT guard are happy with the site, and they’re furious with each other.

The autopsy for this tech fiasco may not be the most contentious software review in history, but it will certainly be the first one driven by Congressional subpoenas.

The problems started early, but for the most part stayed hidden. In June, the GAO publicly warned that much of the work on the state and federal exchanges wasn’t finished, that many deadlines had been missed and that there was no guarantee exchanges would run smoothly by October 1. But the many contractors said the site would be ready.

The guidelines kept changing. There wasn’t enough time to test. Every company was confident about the piece it was building, but some had stopped communicating with the other contractors. Even CGI couldn’t get the tools it needed to run proper diagnostics. CGI first logged this problem in June, but it persisted through September.

Meanwhile, CGI’s Cheryl Campbell testified to the House Energy and Commerce Committee that it had completed an operational readiness review and that its part of the exchanges would be ready on time. QSSI, Serco (which manages paper applications), and credit agency Equifax all told the committee the exchanges would be ready.

Development Seed proudly showed how its website would be built from state-of-the-art open-source technology that it would enhance and then release back to the developer community via GitHub; testing with users would continue beyond the launch, and the site, it said, would continue to improve. Health and Human Services was also pleased with its new site: Chief Technology Officer Bryan Sivak told The Atlantic’s Alex Howard that it was “fast, built in static HTML, completely scalable and secure.” It was hoped the site’s design and speed would save money and attract younger users, on multiple devices, in multiple languages, creating a shining new standard for innovative government IT projects.

Behind the scenes, contractors and subcontractors began placing worried calls to each other. Everyone was frustrated with CMS’s inability to corral the many companies it had to pull together. Each had its own story of bad data sets, broken application programming interfaces, information or access it needed from other companies. “This is a s**t-show,” some said to each other privately. No one was certain whether the site would work, which parts would break first, or even whether they would quickly be able to fix what was broken.

QSSI warned the Obama administration that there were serious problems just weeks before the launch. But the law mandated that the exchanges be opened on October 1, and everyone was determined to have the web exchanges open too.

From a user perspective, one of the troubling things about is that when the site’s back end can’t find the data it needs, the site fails silently. The user isn’t sure whether it’s an Internet connection problem or whether he has accidentally keyed in the wrong information. The sites maintain their beautiful, inscrutable masks, while everything behind them bangs together and falls to pieces.

This isn’t limited to This week, when one of the data centers powering QSSI’s vaunted data hubs failed, all of the exchange sites – state and federal – stopped working. The only thing site operators can do, once they realize there’s a failure somewhere up the stack, is close the entire site.

And despite the Obama administration’s promise of a “tech surge” and the appointment of former OMB official Jeffrey Zients to get the system fully functional by the end of November, there is no cavalry on the horizon, no armies of engineers who can wade in to fix what’s been broken.

Three things are likely to happen by the end of next month. The site will gradually begin to mostly work. There will be serious calls to do what the administration absolutely did not want to do: extend the deadline for the individual mandate to account for the fact that it has been difficult for many people to sign up for plans. And the Energy Committee hearings, which are purely for show, will give way to the House Oversight Committee, which will subpoena everyone involved to find out what happened. The American people will be given the chance to learn more about IT procurement, web development and client-and-server requests than it ever wanted to know.

The U.S. government built these websites in its own image: a potent and destructive stew of idealism and cynicism, the past and the future, technical prowess and bureaucratic bungling, lies, and reality. shows that Americans are a people of great wealth and great need. The U.S. is a surveillance state and a welfare state, and yet it is slowly and painfully trying to build something that can reach beyond the nation’s squabbles and inertia.

It’s a worthy goal, but not one that can be reached through bad compromises and broken software.