When the deadline hit last week on the official round of public feedback as the Federal Communication Commission makes rules on so-called net neutrality, it triggered a tsunami of online responses. When all was said and done, 3.7 million comments had been recorded by the federal government, more than the FCC has gotten on any debate in its 80-year history.
All were filtered through an obscure but critically important piece of software called the Electronic Comment Filing System, the commission’s online system for public input that is, remarkably, nearly the same as it was when it debuted 17 years ago. But that flood of public interest revealed just how dated the FCC's online public engagement infrastructure has become.
As the system strained under the attention, grass-roots activists and staffers inside the FCC worked together, hour-by-hour, to keep it up and running. At times they got on each others nerves; at others they pulled together. But all were coping with the same decades-old technology that, no one can now deny, simply isn’t up to the age of digital civic engagement. The FCC online commenting system's sputtering was an embarrassment to an agency eager to prove it is competent enough to make rules around modern technology and a frustration for groups with names like Fight for the Future and Demand Progress that are eager to prove both their organizing chops and the rightness of their issue.
That we are seeing an increasingly networked, politically engaged public run smack into the limitations of government IT isn't news to those charged with running the technology. For the last year, David Bray has been the FCC's chief information officer. Style-wise, the blue-and-gold online filing Web site looks behind the times. Dig deeper, Bray says, and it only gets worse.
“Being a federal CIO,” Bray says, “means buying a used car, driving it off the lot, and only then looking under the hood and seeing what engine you’ve bought. And I’ve got a pretty old engine.”
And yet, the number that would eventually spit out of that creaky online machine mattered a great deal to those trying to show how invested the mainstream American public had become in the net neutrality debate. Holmes Wilson is the co-founder and co-director of Fight for the Future, one of the handful of networked activist groups leading the charge.
“The final count was extremely important to us,” Wilson says. “When you have a protest, the first thing anyone says is, ‘How many people came?’”
When the digital dam broke
Maintained by the FCC's Consumer and Governmental Affairs Bureau, the Electronic Comment Filing System has admirable goals: bringing the public into the work of government while creating a living record of that engagement. It was, at its moment of creation, cutting-edge. But today it’s the poster child for what happens when a government limping along on 20th-century technology meets the expectations of a thoroughly digitally-connected 21st-century public.
The digital filing system has had its trials since net neutrality debate formally began at the agency on May 15. On June 2, a video segment from comedian John Oliver that called on the public to contact the agency began going viral. The FCC's system froze, but not from traffic load. Data later released by the agency showed that after the Oliver piece the number of comments that flowed in numbered only in the tens of thousands -- some 9,700 the day it aired, and 14,900, 16,500, 13,800, and 24,000 the following days. Instead, the problem was so-called record locking, or what happens when a commenter -- or bot -- files comments and immediately searches for them. The dozen or so technologists in the FCC's tech shop, Bray would later say, began "MacGyvering," figuring out that a slight delay between accepting and publicly posting the comments would do the trick.
Search became a problem again around the July 15 intermediate deadline on the first round of comments. Rumors circulated that some comments were missing from the system. There was a rush of search requests on the service; that's a problem, because the site is largely search-based. Bray and his team introduced a cached search index. With that patch in place, the system stayed afloat even as hundreds of thousands of comments poured in.
Facts notwithstanding, Oliver would get widespread credit for "crashing" the FCC Web site. The ability to funnel large numbers of people through online channels of civic engagement has become a sign of tangible power. Marvin Ammori is a lawyer and activist who sits on the board of some of the advocacy groups involved in the open Internet fight.
“It’s a political question," says Ammori, "how you count these numbers.”
That's one reason why, as the Sept. 15 deadline approached, net neutrality activists scattered across the country were keeping up a round-the-clock watch of ECFS's health. Headquartered in Southwest Washington, D.C., the FCC's technologists were doing the same.
Laying a pipeline into the federal apparatus
A trail of e-mails that shot back and forth between activists and the agency around that mid-September deadline revealed an unusual collaboration. All involved saw the deluge coming, and activists -- planning a one-day "Internet Slowdown" that called on the public to contact the FCC -- reached out to the bureaucrats to see how they might help keep the system afloat.
There were early signs of trouble. “You’d write some very long thing and hit submit,” Wilson says of the average commenter’s user experience on FCC.gov, “and it would spin for five minutes and then show you some incomprehensible error message.”
It's worth noting that the activists had decided to avoid directing would-be commenters to OpenInternet@FCC.gov, the e-mail address the bureaucracy had set up as an alternative input mechanism. As a bit of a make-do measure, e-mails were handled in a quirky way. They were counted one by one as they came into an agency inbox and were added individually to the official tally. But they were uploaded to EFCS in large PDF batches, and so thus appeared online, unofficially, as only one record.
No, the activists wanted a direct line into the vein of the comment tallying machine. So they built clever tools, one of which was hosted on a site called Battle for the Net and pushed out to users across the Web. The trick to it was if the FCC Web site spun out and the comment didn’t register in the system, the activists platform would save a copy and attempt to refile it later. (Observers point out that the technique gave the activists another useful opportunity: the ability to capture e-mail address to add to their mailing lists.)
And so, on Sept. 10, the activists saved up a batch of some 100,000 comments and then tried to file them through ECFS. But the system was having trouble accepting them.
At 3:45 p.m., the activists sent the agency officials a note: “We noticed the FCC site was having issues, so we paused submission of comments. We’ll try turning it back on a bit later as things quiet down.”
At 3:57 p.m., the FCC replied: "We turned off conversion – the volume level of ECFS comments had recently gone up starting at 3 p.m. and was straining the submissions by causing database deadlocks. If you want to continue at the volume level you were doing around 11 a.m., we’ll let you know if any issues return?"
There was more back and forth, and shortly after 7 p.m. the FCC said that it was working on a fix. The issue, upon investigation, seemed to be that each filed comment creates a unique connection to the database, one that automatically times-out in short order. But a batch of, say, 25,000 comments at once was creating 25,000 connections that seemed to persist. In response, the FCC team set up an alert that let one of their technologists know when a thick cord of database connections needed to be manually snipped.
At 10:10 p.m., agency staff again e-mailed the activists: "You can start uploading comments again. Thank you for the patience.”
The activists began transferring big quantities of comments yet again. But by a 11:45 p.m. they realized that they had a backlog of half a million comments; they wouldn’t be done with the moving process in time to count for that day of action’s tally and the inevitable round of press attention.
So they sent a note to the FCC aides letting them know that they were backed up and couldn't get all their comments in by the daily deadline.
Dave Karpf is an assistant professor in the School of Media and Public Affairs at the George Washington University and author of the 2012 book “The MoveOn Effect: The Unexpected Transformation of American Political Advocacy.” What's striking in this case, says Karpf, is that "this particular politicized public has workarounds."
"If I was still at the Sierra Club,” where he worked before becoming an academic, “and this was a climate issue, we probably would have just thrown up our hands and issued a press release."
Searching for workarounds, on and off the federal payroll
The agency technologists scrambled for a way to move ahead. After vetting the proposal with the commission’s lawyers, on Sept. 11, the day after Internet Slowdown Day, they posted to the agency’s blog on FCC.gov instructions for filing comments in giant batches by converting them into Comma Separated Values, or CSV, file format and then shipping those digital packages along to the agency.
On the backend, the technologists then set up an automated process for finding e-mails with, per the instructions, “CSV” in the subject line and moving them directly to a Box.com online cloud storage account. (That introduced an extra step, but the agency aides were reluctant to simply open up a digital public dropbox, as anything could be put in there, including pornography. The “CSV” subject line test wasn’t much of a preventative device, but at least this way the agency would have an e-mail address on record.)
While the agency was piecing together those measures, the advocates saw that overnight their backlog had grown to three-quarters of a million comments. But by midnight, only a fraction had been formally filed into ECFS. So they asked the agency to pass along any press inquiries to the advocacy groups so that they could relay their unofficial but accurate count.
But press management is an imperfect science, and word that the total number of file comments was an underwhelming 100,000 found its way out of the FCC press shop. “That was wrong,” Ammori, the activist-lawyer, says of the tally. “Very wrong.” That figure, though, was printed and reprinted widely on Thursday, including — to the particular annoyance of the organizers — in the New Yorker. Feeling that they’d held back filing some of their batched comments because of the FCC’s guidance, the advocates were dismayed.
Due to high number of comments received in last 24 hours, we do not yet have an official comment count. Will provide update ASAP.
— Kim Hart (@khart) September 11, 2014
To ease minds and soothe tensions, the advocates and bureaucrats worked together to correct the record. Shortly after 1 p.m. Sept. 11, the FCC press secretary sent out a carefully crafted tweet: “Due to high number of comments received in last 24 hours, we do not yet have an official comment count. Will provide update ASAP.”
The activists kept up their bulk submission of comments over the weekend, and by the following Monday, all were in. Near the end of the day on Tuesday, Sept. 16, the FCC spokesperson sent reporters a note with a final, official tally: some 3.7 million comments. A whopping 828,000 came in during just the last few days of the activists’ push.
State-of-the-art back when Bill Clinton was president
From the beginning, the Electronic Comment Filing System was aimed at using the Internet to level the playing field at an agency that has never had trouble hearing from big business and their lawyers but has sometimes struggled to connect with the general public.
Kevin Werbach was policy counsel at the FCC in the 1990s. The American public, he says now, was increasingly using the Internet to interact, but anyone who wanted to see what was being said to the agency in the course of telecommunications debates had to trudge down to a single document headquarters and pay for copies. They started with an e-mail inbox, but even then it required too much administrative wrangling.
So Werbach and others extracted a bill of the supplemental funding from the landmark 1996 Telecommunications Act -- “I believe it was $200,000 to $300,000,” recalls Werbach — and set up a Web-based system called QuickStart. But that first system only went half-way; it was a physical-digital hybrid. Commenters sent in their filings on diskette, the contents of which were eventually posted online.
The agency soon began a push to make the process fully virtual and to get the digital comments to count as part of the official record. That expansion, the agency noted optimistically, nearly two decades ago, “will help the FCC gain experience in managing the workflow of receiving comments in electronic format, and will also give the public an opportunity to try out the new system.” (The FCC’s records from this time reveal both an amazing amount of foresight and an overabundance of faith in the agency's technological capacities: “Although comments filed via the Internet may take time in transit to the FCC due to network congestion or large attached files,” the commission wrote in April of 1997, “we believe that this transmission period will usually be quite short.”)
For the late '90s, these forays into digital governance qualified the FCC as an early adopter. The thing is, ECFS hasn't changed much since. Says Werbach, "How many major Internet applications from the mid-'90s, before Google was even founded, are still going strong?” Still, he says, it needs an upgrade.
In the absence of funding, a cobbled-together fix
Today the FCC is a communications agency that can’t communicate, and that irony is not lost on those inside the agency. “It is particularly distasteful that the FCC – the agency entrusted with promoting a world-class broadband infrastructure for the nation – could ever be incapable of dealing with Americans expressing themselves via that broadband capability,” FCC Chairman Tom Wheeler said in a blog post from mid-July,
Reed Hundt was the agency's chairman from 1993 to 1997. It was on his watch that ECFS was created, and he's encouraging his successor to see the historic level of public engagement as an opportunity to more fully embrace that feedback loop. "Any time an FCC chairman can get any level of attention from millions of people," Hundt says, “it’s fundamentally a great opportunity for the chairman and a good day for democracy."
But even just rebuilding a working Electronic Comment Filing System is no simple matter. The ECFS that the public sees is only the tip of an iceberg; on the backend, the system is tied into seven databases that are used by 18 offices within the FCC. What’s more, ECFS is just one of 200 often-dated IT systems in use inside the agency — or one for every nine of the agency’s 1,750 employees. “Relic systems,” Wheeler called them when he went before the House Appropriations Committee for the agency’s budget approval in March.
The rub? These systems, testified Wheeler, are “costing the agency more to service than they would to replace over the long term.”
Congress, however, has a history of balking at smart long-term investments. To fund the agency next year, Wheeler has asked for some $376 million. Of that, $9.2 million is allocated for the “modernization of aging IT systems.” There’s also a pool of money to hire 10 more technologists; at the moment, the agency has 36 — or about half of what it says an agency of its size should have.
House appropriators cut some $53 million from that request, dropping the agency below even what it's on track to spend this year. If the budget passes, Wheeler will face tough choices about where to spend the commission’s funds.
But those in the FCC tech shop aren’t waiting. Under CIO Bray’s direction, the team is whipping together a stripped-down reboot of the electronic comment filing system. The new software will capitalize on what the technology world has figured out in the last 20 years about making digital systems more resilient — such as cloud computing (definitely) or CAPTCHA tests (possibly) to make sure submitters are human. The plan is to get that stop-gap system in place by the end of the year.
It’s not a solution. But it is a fix. And at the moment, that seems to be the best anyone can do.
Have more to say about this topic? We take your questions every week in our weekly livechat, Switchback, Fridays at 11 a.m. ET. The comment box is open, so submit your questions now.