As we learn more about what went wrong with the design and launch of Healthcare.gov, a few broad principles have emerged about how to fix the procurement system so this kind of debacle -- which isn't the only non-functional Web site the government's bought, just the highest profile -- doesn't happen again.
One of them is the willingness to entertain innovative proposals from companies that might not have years of federal contracting experience. As we mentioned last week, CGI Federal landed the Healthcare.gov contract as a task order, having been certified four years previously to compete with a select group of other big firms for anything the Department of Health and Human Services might need. That keeps out newer firms that might have a more novel approach to the problem, as well as those that just don't feel like dealing with jumping through all those government hoops.
Another is the openness of the software development process: While the front end of the Web site was developed open source, CGI Federal's backend was entirely secret, which meant that coders couldn't crawl around and help spot bugs. As Bloomberg Businessweek outlined, posting the code on Github would have brought a community of people out to fix problems before they got so deeply embedded that there was no way to extract them. It's standard in Silicon Valley, and even for lots of major governments around the world as well. The Obama administration even used it for an earlier version of HealthCare.gov.
A third, however, goes to the very heart of how companies build large software systems: The federal government's been doing it backwards for decades.
To understand why, you need a little history. Government contracting culture and rules originate with the Department of Defense, which knows what it wants ahead of time, and can draw exact specifications that companies compete to fulfill. A tank is a tank is a tank, pretty much, plus or minus a few bells and whistles.
The software development version of that is what's known as the "waterfall" model, where bidders describe what they're going to do, and then proceed to design and construct it -- with testing all the way at the end. A big problem with Healthcare.gov was that the team didn't have time to run many tests before the Oct. 1 cutoff date, which meant nobody knew quite what would happen when it went live.
Larry Fitzpatrick, who used to run IT systems for the Financial Industry Regulatory Authority and now serves as president of a Bethesda-based tech contracting firm, says that makes no sense.
"That style of development is something that the government continues to latch onto, because it seems to be easy to understand, easier to procure for. But it generates a tremendous amount of risk," he says. "What often happens when you have these big requirements up front, is the people who are specifying the product are afraid of not getting all their ideas in, so they overscope the project. And then the development team is on the hook for delivering everything, not just the essential elements."
The better way to do things is a school of software development called Agile -- it's been around since the 1950s, was basically codified in the early 2000s, now has a whole non-profit devoted to it, and is the dominant form of software design in teams. Rather than moving from one static stage to the next, it emphasizes constant iteration and testing, with prototypes building on prototypes so the endpoint is something that works. The only problem, from a government perspective, is that you need to be comfortable with not knowing exactly that they will look like.
"If you're going to take the time to write a 1,000 page specification, you're going to miss the point," Fitzpatrick says. "They should have fairly quickly, within a quarter, been able to put together a skeleton for what the system had to do, what systems had to talk to each other, and get it up and running. This is really the only way to build systems of this complexity."
It's not as if nobody in the government knows how to do this. Fitzpatrick says his firm, Computech, used it to put together the National Telecommunications and Information Administration's national broadband map, and that the Customs and Immigration Service has used agile principles for some of its projects as well. They tend to fly under the radar because they're less expensive projects -- but they're also less expensive because they don't require massive numbers of people to put together. [UPDATE: As Suzy Khimm notes, the Consumer Financial Protection Bureau has done a lot of this as well, in part because it has a robust in-house tech shop]
"I mean that's an alarm bell," Fitzpatrick says. "If you have a software development project that costs more than $40 to 50 million, you really have to ask why."
So how does agile become the standard mode of software development for all federal IT contracting? It's more a cultural shift than anything else, said a bunch of federal tech people at a panel on the topic last year. Project managers need to get used to the idea of working intensely with a development team rather than asking for a specific thing and then walking away -- as Office of Management and Budget policy analyst Tim McCrosson put it, "if we can get out of that mindset that we're buying a product and get into the mindset of we're buying a process."