The resulting implosion of the app from Shadow Inc. on the first vote of 2020 undermined faith in the electoral process just as Democratic party leaders were trying to restore it. The country is still waiting for the full results.
“The most important lesson anyone should take away from this is that if you're going to use a new technology that you need to very rigorously test it and exercise it and plan for what your backup will be if it fails,” Eric Rosenbach, a former top Pentagon official who leads the Defending Digital Democracy program at Harvard University, told me.
“That didn’t happen here, which is disappointing," he said. "That’s not something that’s good for democracy.”
The high-profile coding error, which produced inconsistencies in reported caucus results, was reminiscent of other times when the government or campaigns built digital tools -- but the hard work of getting the tech right took a backseat to other priorities.
In 2013, health insurance seekers could barely use the Obama administration’s online exchange Healthcare.gov when it was first released, dealing a serious blow to the former president’s main domestic policy achievement. It prompted jabs from Republican critics that a government that couldn’t build a website couldn’t possibly revamp healthcare that some echoed this week.
It took two months and the intervention of a tech “tiger team” before the site was functional.
And on Election Day 2012, the Mitt Romney campaign outfitted thousands of volunteers with a custom app called Orca that was designed to share real-time analytics from polling places and get-out-the-vote efforts in swing states. It was supposed to mark a revolution in campaign technology and a Republican counterstrike to the 2008 Obama campaign's Project Houdini program, which helped deliver one of the most data-driven Election Day operations in history. Instead, the app failed to load for many volunteers and crashed repeatedly for others, leaving many volunteers with nothing to do.
The flops came as top leaders tried to meet political timelines leading to a major failure on the national stage. The Shadow app seems to have been built in an even hastier fashion.
The Iowa Democratic Party only opted to build an app for reporting caucus results late last year after the national party raised security concerns about reporting results by phone, the New York Times’s Matthew Rosenberg, Nick Corasaniti, Sheera Frenkel and Nicole Perlroth reported. Shadow, which they reported won the contract over other multiple bidders, built the app in just two months, which is almost certainly far less time than major tech companies would take.
That rushed approach often spells disaster for tech projects, which need extensive testing to ensure all the major bugs are worked out, Steven VanRoekel, a former Microsoft executive who was the nation’s chief information officer during the Healthcare.gov launch told me.
“It seems as if they tried to deploy something last minutes that hadn't been vetted, hadn't run through rigorous testing and that's when stuff breaks down,” he said.
The Iowa party was also focused on dozens of other caucus priorities and seemed to be ignoring a lot of warning signs when it came to the app. That echoes the initial Healthcare.gov release when most of the people responsible for it in government had other important jobs and no one was laser focused on ensuring the technology worked, VanRoekel told me.
“If it's not a wake up everyday and do it project for someone, you shouldn't ship it,” he said. “You shouldn't put it out there because it's just destined for failure.”
On the company's side, Shadow was also hampered by a lack of coding expertise, an unfamiliarity with major tech projects and not enough time to get the app approved by Apple’s app store, the Times reported. "Instead, the app had to be downloaded by bypassing a phone’s security settings, a complicated process for anyone unfamiliar with the intricacies of mobile operating systems, and especially hard for many of the older, less tech-savvy caucus chairs in Iowa," they write.
The massive delay in the results on such a high-stakes night not only diminished public faith in Democrats' ability to run elections but created a situation ripe for disinformation, tainting the caucus as much as a foreign interference campaign might have.
“The danger of a snafu like this is that it undermines trust in the democratic process and the more often that happens the less likely Americans are to trust that the vote they put in is actually the vote that counts,” said Rosenbach, whose group focuses on combating foreign interference in campaigns. “They're more likely to start to believe some of the fringe conspiracy theories that pop up on Twitter or on Facebook.”
State election officials, who have spent months or years vetting voting systems and other technology before they're approved for use in general elections, described Iowa's situation as "cringeworthy." State election officials often don't play a role in technology used in caucuses, though they typically supply state-approved machines in states with primaries.
"Before you can make any technological change in elections, you have to have two things. You have to have confidence and you have to have competence,” Ohio’s Republican Secretary of State Frank LaRose told my colleague Neena Satija. “What I saw was kind of a cringeworthy thing for any of us that run elections ...But they put themselves in this scenario by what sounds like a series of bad decisions that they had made.”
There were plenty of warnings.
Congressional staff and computer science experts raised doubts for months about whether the app would work -- but their questions were rebuffed by Democratic leaders, my colleagues Tony Romm, Neena and Drew Harwell reported.
The Department of Homeland Security also offered to test the app's security but was turned down, acting secretary Chad Wolf said on Fox and Friends. Iowa Democratic Party chair Troy Price later said he didn't know about the offer.
The Democratic National Committee's chief security officer, Bob Lord, even urged the state to drop the app but was ignored, sources told the Wall Street Journal’s Dustin Volz, Tarini Parti, Alexa Corse and Robert McMillan.
For its part, the party cited security as one reason it sought to avoid the spotlight in the runup to Iowa, and said the delay in testing until just weeks before the caucuses was intentional. "Iowa Democrats waited to introduce the software to avoid giving potential hackers time to penetrate it, the official said, adding that the party chose to keep the name of the vendor secret on the advice of national cybersecurity consultants," my colleagues Isaac Stanley-Becker and Michael Scherer report.
Still, by the day after the debacle, officials were also quick to denounce the app. Democratic officials in Nevada, who’d planned to use it for their own caucus Feb. 22, quickly reversed course. DNC chair Tom Perez declared the app “will not be used in Nevada or anywhere else during the primary election process,” adding that “the technology vendor must provide absolute transparent accounting of what went wrong."
Price apologized for the delay when he released partial results yesterday afternoon but also stressed that none of the results were hacked or tampered with. “The underlying data, the raw data is secure,” he said. “It was always secure. This was a coding error.”
Shadow itself, meanwhile, acknowledged the coding error that prevented caucus leaders from sharing results and issued an apology on Twitter:
The company's chief executive Gerard Niemera, who previously built tech products for field organizers in Clinton’s 2016 campaign, also offered a personal apology in a Bloomberg interview. “I’m really disappointed that some of our technology created an issue that made the caucus difficult,” he said. “We feel really terrible about that.”
PINGED, PATCHED, PWNED
PINGED: As Democrats grew impatient for caucus results Monday night, Rosenbach’s group Defending Digital Democracy -- which was built in part to combat misinformation campaigns and conspiracy theories – itself became the focus of a false rumor spread online.
Hundreds of people on Twitter began claiming that Defending Digital Democracy had built the malfunctioning caucus app – possibly because they’d learned the app had connections with Clinton’s 2016 campaign. One of Rosenbach’s co-founders is Clinton’s 2016 campaign manager Robby Mook. Mitt Romney’s 2012 campaign manager Matt Rhoades was also a co-founder of the bipartisan group.
The claim had no basis in reality, but it exemplified how a phony narrative can take hold -- especially when facts are hazy – and do real damage to the democratic process, Rosenbach told me.
“It’s a near perfect example of how misinformation, disinformation and a little bit of bad and irresponsible reporting turns into something that takes over parts of the Twittersphere in a way that then contributes to undermining trust in the caucuses and in democracy,” he said.
There’s no evidence that the false rumor was spread or amplified by a foreign adversary or anyone other than legitimately confused people online – though many spreading it were Clinton critics. Rosenbach's group is considering how they might use the experience as a lesson in future education programs about disinformation, he told me.
“You see it spreading from Twitter and then, exactly like the Russians would do it, pumping it onto mainstream media,” he said. “This afternoon, literally, we've had three dozen phone calls from people who want to know why we developed the app, how we developed it, why was Harvard doing that? It’s really scary to see the arc of all that develop.”
PATCHED: The White House is working with U.S. tech companies to create 5G software that would help the United States and its allies reduce their reliance on Huawei telecommunications equipment, Bob Davis and Drew FitzGerald at the Wall Street Journal report.
The plan would require U.S. telecom and technology companies to agree on standards that would allow 5G software to work on any equipment. Companies including Microsoft, Dell and AT&T are part of the effort, White House economic adviser Larry Kudlow told Bob and Drew.
Unseating Huawei, which U.S. officials say is too cozy with the Chinese government, won't be easy. The United States arrives late to the game and the initiative could threaten non-Chinese competitors such as Nokia and Ericsson. There could also be complications in coordinating with European allies to build the technology, Bob and Drew report. The United Kingdom recently ignored U.S. urging to ban Huawei from building its 5G network, but Kudlow told the Journal he hoped the new project could lead to a reversal.
Lawmakers have also pushed for the U.S. government to help the private sector compete with the massive subsidies Huawei offers China. The White House has not decided whether it will back the effort, Kudlow says.
Meanwhile the White House plans to bring together high-level officials from the Commerce, Defense, State and Energy Departments later this month to discuss potential further limits on U.S. companies selling to Huawei, Reuters reports.
PWNED: The Homeland Security Department is doing a good job developing mandatory cybersecurity rules for government agencies but still lags when it comes to making sure agencies are following those rules, a new government watchdog report finds.
“We found that these directives have often been effective in strengthening federal cybersecurity,” a Government Accountability Office report notes. “However, agencies and DHS didn’t always complete the directives’ actions on time.”
For instance, DHS mandates that agencies patch computer bugs within 30 days. But only 61 percent of vulnerabilities were mitigated within that deadline in 2019, a more than 20 percent decrease from the year before.
The report also dinged DHS for failing to fully protect “the government's most critical information and system assets.” A DHS directive requires the agency to assess all high-value government agency assets, but DHS completed only about half of those assessments for the past two years.
DHS also has yet to issue guidance for security contractors working with those high-value assets such as computer systems that contain citizens’ personal information. The agency plans to do so later this year, it said.
DHS agreed with the GAO recommendations, including that it should develop a way to independently verify that agencies are making the fixes they say they are.
— Cybersecurity news from the public sector:
— Cybersecurity news from the private sector:
THE NEW WILD WEST
— Cybersecurity news from abroad:
- New America will host an event on "Kickstarting the Digital Heartland" Wednesday from 12 p.m. to 2 p.m.
- Georgetown Law’s Institute for Technology Law & Policy in partnership with the Georgetown Law Technology Review will co-host a daylong conference on “Election Integrity in the Networked Information Era on Friday from 9 a.m. to 5 p.m.
- The Senate Homeland Security and Governmental Affairs Committee will host hearings to examine a roadmap for effective cybersecurity, focusing on what states, locals, and the business community should know and do on Tuesday at 9:30 a.m.
- The Senate Committee on the Judiciary, Subcommittee on Intellectual Property has scheduled a hearing entitled “The Digital Millennium Copyright Act at 22: What is it, why was it enacted, and where are we now” for Tuesday at 2:30 p.m
- The Senate Armed Services Committee will host hearings to examine United States Special Operations Command and United States Cyber Command in review of the Defense Authorization Request for fiscal year 2021 and the Future Years Defense Program on Thursday at 10 a.m.
- RSA Conference 2020 is scheduled for Feb. 24 to 28 in San Francisco