DES MOINES — Six hours before Iowa's first-in-the-nation presidential caucuses were set to begin, an engineer for the Democratic National Committee offered a frank appraisal of the software that would be used to verify the results collected by hundreds of volunteer precinct leaders across the state.

“I’ve tested this as well as I can without real data but no guarantees it . . . won’t all come crashing down once real data comes in,” he wrote in a private channel on Slack, the online messaging platform, according to records viewed by The Washington Post.

The warning was prescient.

About seven hours later, leaders of the national party’s tech team burst through the doors of a conference room in the Iowa Events Center, where Troy Price, the chairman of the state party, was gathered with his staff to oversee the operation.

There was a bug in the software, they told him, according to multiple people present.

That’s when it all came crashing down.

No results were published until the following afternoon. When the numbers trickled out, they were riddled with obvious inaccuracies.

The first-in-the-nation voting state was thrown into chaos on Feb. 3 after the Iowa Democratic Party delayed releasing results. (The Washington Post)

By the end of the week, an effort was underway to place blame squarely on the state party, which critics portrayed as ill-equipped and unprepared for its moment in the spotlight. The chairman of the DNC, Tom Perez, went on prime-time television to call the caucuses a “major league failure,” saying of the Iowa chairman, “he owned up to it.” Perez called on the state party to begin a hand audit, tweeting: “Enough is enough.”

Six days later, Price, 39, once a rising star in Iowa politics, announced his intention to step down, declaring that he was “deeply sorry for what happened.”

But a detailed review by The Post found that the chaotic events of Feb. 3 were years in the making, and that the responsibility extends beyond the local party leaders who have borne the brunt of the criticism.

Rather, the turmoil in Iowa reflected a systemic failure in which Democratic officials eager to avoid repeating the disastrous campaign of 2016 — marred by Russian hacks of party emails and allegations that the nominating process had been tilted against Sen. Bernie Sanders (I-Vt.) — oversaw changes that triggered a whole new crisis.

At the heart of the reforms would be a newly transparent caucus process, with raw vote totals announced publicly for the first time.

Party officials, however, never effectively vetted the basic tool used to collect and publish those results, the review found. They hardly questioned why an app was necessary, rather than a simpler reporting method, though internal correspondence shows that DNC staffers were privy to discussions about the testing and rollout of the technology.

This account of the debacle in Iowa — the actions that caused it and the fallout that cast a cloud over the early days of the 2020 election — is based on an examination of internal documents and correspondence related to caucus planning as well as interviews with 54 people who were involved, some of whom spoke on the condition of anonymity because they feared for their future in Democratic politics.

A spokeswoman for the DNC, Xochitl Hinojosa, maintained that the national party’s role was limited to ensuring the cybersecurity of the software, which was built by veterans of Hillary Clinton’s presidential campaign at a start-up called Shadow Inc.

She said the Slack messages showed only that DNC tech staff was brought in to “create a backup system to double check the delegate math from the app . . . as a precaution in case there was a hack.” And she said their work had “nothing to do with the app code or the coding issue in the app that caused reporting delays.”

Shadow apologized for its error, which it said did not affect the integrity of the underlying results.

But the episode has called into question the broader credibility of a process whose importance in determining American presidents has taken on epic significance since assuming the first-in-the-nation status in 1972. Multiple people with years-long involvement in the caucuses said the mathematical irregularities exposed by new transparency rules point to problems that have often been present but quietly glossed over by party leaders before final results were announced.

The delay in results from Iowa’s first-in-the-nation caucuses Feb. 3 was due in part to problems with a mobile app used to report figures to the state party. (The Washington Post)

Just as Sanders won the popular vote this year but came in second in official results due to the process of delegate allocation, multiple people involved in 2016 said the same outcome likely occurred then. There was simply no public tally to prove it.

“We live in a different era, where people expect certainty and they expect it instantaneously,” said John Norris, a longtime political operative and activist in Iowa. There was a “failure,” he added, “to plan for complete failure.”

The party’s caucus woes may not be over — and not just because Sanders and former South Bend, Ind., mayor Pete Buttigieg, who claimed most delegates in Iowa, are both asking the Iowa state party to audit the results.

Democratic officials in Nevada, who were to use two apps built by Shadow for their caucuses this month, are now racing to avoid a similar failure. Party leaders scrapped the software — though they had conducted earlier and more rigorous testing than was completed in Iowa, according to strategists involved in the two states — and devised a last-minute replacement in consultation with the DNC.

Iowa, however, continues to wrestle with the fallout.

Dozens of state and local Democrats interviewed for this story expressed dismay because, inside hundreds of precincts, the caucuses ran smoothly. What came unglued was technology out of their control — turning this quadrennial institution of U.S. politics into a national punchline, ridiculed by comedian Steve Martin at the Oscars as well as by President Trump’s campaign manager, Brad Parscale, who described the process as the “sloppiest train wreck in history.”

“Talking heads didn’t get to say what they wanted to say when they wanted to say it,” said Rob Sand, the 37-year-old state auditor, defending the caucuses as an integral part of the “social fabric.”

Saving the caucuses

The roots of the fiasco in Iowa lie in the bitterness that followed the 2016 campaign.

So fractious was the Iowa party that Price, who had previously been its executive director and then a top Iowa strategist for Clinton’s 2016 campaign, initially decided not to run for the chairmanship. He was worried about the animosity toward people associated with Clinton, whose razor-thin victory over Sanders led to accusations of bias, even of cheating.

National party officials, desperate to restore confidence in the nominating process, required states to make their caucuses more accessible or switch to primaries, as a handful of states did. The party also put out new rules requiring the calculation and publication of three figures on caucus night, including two different expressions of the popular vote.

Price, who decided to seek the chairmanship only after the previous party leader left for medical reasons, worked to unite warring factions of the party.

Dealing with the DNC, and its new rules, turned out to be the most difficult part of the job, he said.

“We were trying to take rules that were clearly different than our system allows,” he said, referring to the mathematical rigidity imposed on what started out as an informal party-building exercise. “For the most part, I think we were successful in doing that.”

Price first floated the idea of allowing people to caucus via teleconference — a system that came to be called “virtual caucuses” — at a DNC meeting in Puerto Rico at the end of 2018, and asked for expedited review from national leaders, he said.

But Price and his team struggled to get answers from the national party or its tech staff, according to multiple operatives involved in early planning, some of whom later left to run campaigns and so spoke on the condition of anonymity because they were not authorized to speak publicly.

Raffi Krikorian, who left as the DNC’s chief technology officer in April 2019, said he couldn’t recall the requests but added it was “entirely possible” they were placed, saying the tech staff hadn’t yet been “roped in on all the caucus-related things.”

Iowa Democrats unveiled their plan in February 2019, beginning a public comment period before the proposals went up the chain to the DNC. At the time, Price predicted that the changes, which would have allowed Democrats to participate by phone in the week leading up to caucus day, would “make the 2020 caucuses the most accessible, the most transparent, the most secure and the most successful caucuses ever.”

Around the same time, the state party began a process of engaging technology companies, including Microsoft, which had built the reporting app used by both Democrats and Republicans in the 2016 caucuses.

A request for proposal went out in May. About a dozen firms responded, according to people involved in the process. Contracts were signed for some parts of the system, but not for reporting results.

Planning the caucuses

An app for reporting results appeared nowhere in the original request, which was obtained by The Post. Microsoft said it wasn’t interested in doing another caucus app, according to state party operatives. Iowa Democrats were stuck.

That’s when Iowa’s caucus team heard about Shadow from top Democrats in Nevada, who were already planning to use the company for reporting software. The executive director of the Nevada Democrats, Alana Mounce, and the company’s chief executive, Gerard Niemira, both worked on the 2016 Clinton campaign. Mounce declined through a spokeswoman to comment.

Shadow was founded as Groundbase in 2017, focusing mainly on texting tools that had a mixed track record in statewide races. In the election in Virginia that year, for example, its untested peer-to-peer texting platform stumbled, according to a Democratic operative involved in overseeing the race who spoke on the condition of anonymity to avoid jeopardizing career prospects. The tool regularly crashed, the person said.

Niemira, the Shadow CEO, said no issues were brought to his attention but that the company was brand new at the time.

The firm rebranded in 2019 with a major investment from Acronym, a nonprofit digital outfit with an affiliated political action committee. And it has provided services, including texting technology, to several Democratic presidential campaigns, according to aides for those campaigns.

As organizers in Iowa were beginning talks with Shadow about a reporting app, the DNC’s Rules and Bylaws Committee, a panel of Democratic officials from across the country, gave conditional approval in June 2019 to plans for the “virtual caucuses.” But they asked for input from the party’s tech team, whose leadership had changed hands in May and expressed concerns about how the technology would function.

The DNC assembled a “red team” of cybersecurity experts to advise on phone-based voting, according to multiple people who either served on the advisory panel or were asked to participate but declined because members were required to sign a nondisclosure agreement.

In late July, the DNC proposed that Iowa party officials adopt language in its vendor contracts giving the national party access to source code and “operational plans” for the software’s use, according to an email obtained by The Post.

Hinojosa, the DNC spokeswoman, said the language was designed to ensure the vendor would “work with the DNC's cyber-security consultant.”

An additional clause added at the DNC’s request absolved the national party of responsibility “for any allegation, claim, action, suit, proceeding or investigation asserted or initiated related to the Software or this Agreement … regardless of the DNC’s actions, inaction, knowledge, or involvement with regards to the Software or this Agreement.”

The company was paid about $63,000 for its services. It had been expected to receive additional payments in the new year, but the status of the money is now uncertain.

When the DNC’s rules committee convened in San Francisco in August 2019, its members signaled they would recommend against the virtual caucuses based on the expert advice — more than six months after Iowa Democrats had unveiled the details. Hinojosa, the DNC spokeswoman, said the timeline was “in line with previous cycles.”

Price was fuming. He remembers the week in San Francisco as the worst of his professional life — until this month’s fiasco.

“I might have to quit,” he recalled thinking at the time.

The day after the full meeting, Price and his associates in Iowa, as well as Democratic leaders from Nevada, were scheduled to see the committee’s leadership again. Price vowed that he was going to be aggressive with Perez, having spent months crafting a plan to fulfill the DNC’s requirements, only to be told that it was unworkable.

They arrived at the Hilton and received a call directing them to Perez’s suite, which looked out over the city’s skyline. Price did not muster the aggression he had promised. But when he realized the virtual caucuses were out of the question, he made a different inquiry, according to three people present at the meeting: “Can we still use a reporting app?”

The three people said he was assured that the reporting app was acceptable. And two of the people present recalled Kat Atwater, the DNC’s deputy chief technology officer, specifically saying the DNC would help work with Shadow to “get it done,” as one person present recalled.

Hinojosa, the DNC spokeswoman, denied that national leaders had given Price the go-ahead. Atwater, she said, “had heard of Shadow and their work but never vouched for them and also mentioned it was also a tight timeline.”

Running the caucuses

The beginning of autumn brought the state party and the national party closer together in planning for the development of the technology.

The DNC ordered and paid for a security audit of the software, which was completed by NCC Group, a Britain-based cybersecurity firm. Out of the review, which also involved threat assessment conducted with the national party as well as officials in Iowa and Nevada, came directions to guard the name of the vendor, and to take the software live as late as possible to prevent it from getting into the hands of hackers, according to multiple people who participated or had knowledge of the exercises.

By early January, campaigns were becoming fretful, asking when they were going to see the app.

The week of the January debate, Atwater traveled to Des Moines to run through incident response plans developed during a November simulation organized by leaders of the Defending Digital Democracy Project at Harvard University’s Belfer Center. She had been invited to participate in conference calls with Shadow since at least Sept. 17, records show. A contract with the state party was signed on Oct. 14, according to a copy obtained by The Post.

Several news reports in January raised questions about the security of the reporting app, though the party remained tight-lipped about who was developing it. A top aide to Sen. Ron Wyden (D-Ore.) sent three emails before the caucuses to Bob Lord, chief security officer at the DNC, asking how to get more information about the software.

“In particular,” the staffer wrote in the initial email on Jan. 14, “we’d like to find out if anyone has audited it.”

The aide, who spoke on the condition of anonymity to discuss private communications, said there was no response.

The Wall Street Journal reported that Lord urged the state party not to use the app, but Price said his last interaction with Lord was at the August meeting in San Francisco, when he got approval to move forward with the reporting software, according to three people present. In response to questions about Lord’s role, Hinojosa, the DNC spokeswoman, said the party’s “data and technology staff relayed concerns related to the use of technology in elections.”

It was only days before the caucuses that volunteer precinct leaders were able to download a test version of the software. The state party set up office hours for precinct leaders stumped by the system to call in with questions.

But the resources were plainly insufficient, and Price, a week after the caucuses, could not answer questions about how many precinct chairs had downloaded the app, or when training on the software had begun.

On the Saturday before Iowa’s caucuses, meanwhile, the state party in Nevada had conducted a dry run with its early caucus reporting app, which incorporated campaigns and used Harry Potter characters instead of candidates, according to a top Democratic official in Nevada who spoke on the condition of anonymity to discuss sensitive matters.

Nothing of the sort was done in Iowa. Across more than 1,750 precincts, only 624 people logged in to the live version of the app, according to the state party. Just 439 people transmitted results via the app.

The others turned to a hotline number that was supposed to serve as a backup. It was staffed by about 50 volunteers in a boiler room at the Iowa Events Center in downtown Des Moines. They grew overwhelmed when caucus chairs began ditching the reporting app, opting for the call-in number instead.

As images of caucus materials featuring the hotline number began to spread on social media, chaos ensued. Volunteers started fielding calls from CNN reporters searching for results, according to multiple people staffing the phones.

The number was posted on 4chan, the online message board, with instructions to “clog the lines.” Tim Gannon, a former Democratic candidate for Iowa’s agriculture secretary, said more than a quarter of the calls he took when he went in to help in the aftermath of the caucuses were from people who announced their support for Trump.

Over a 24-hour period, 8,000 calls were placed to the boiler room, whose expected capacity was 2,000 incoming calls. Volunteers were able to answer 4,300 of them.

Crashing the caucuses

Price was outside having a cigarette when caucuses convened across the state at 7 p.m.

He returned to his second-floor conference room and waited for results. They didn’t come.

Shortly after 8 p.m., leaders of the DNC’s tech team came through the doors and announced there was a bug in the transmission of results. His mind raced with possibilities, including a potential hack. He was assured that the raw numbers were authentic, he said; they just couldn’t be accurately produced.

Every 15 minutes, he received an update saying the team needed another 15 minutes. On the Slack channel, which had been created by a DNC engineer on Jan. 31, the national party’s tech staff coordinated as they raced to verify the integrity of the results coming through the system. Also in the conversation were top Shadow executives and one representative from the state party.

A chart mapping how data would travel from caucus sites to the results board, bearing the DNC’s logo and obtained by The Post, included checkpoints jointly assigned to the national and state parties. But Hinojosa, the DNC spokeswoman, said, “The results were always in the IDP’s possession.”

The entire setup became inoperative as party officials decided they would have to tabulate results from caucus math worksheets, using Google Forms hooked up via Ethernet rather than the numbers flowing into the database — part of a redundancy plan in the event of a breakdown.

Party officials huddled and decided not to release any results that night. Price and a top aide each held a conference call with the campaigns. Both conversations ended abruptly — the second when a Sanders aide suggested the state party had been riding roughshod over the numbers for years, according to a participant.

Volunteers started placing outgoing calls to caucus leaders to gather their results, and the state party dispatched people all over the state beginning the next day to collect physical records. Some were dropped in the mail.

Meanwhile, errors crept into the figures, which exhibited obvious problems. For example, delegates were assigned to candidates who had too few people in their corner — or, in some cases, no people — in the final expression of preferences. In other cases, the number of caucus-goers increased between the first and second alignment, which should not be possible.

Veteran strategists in the state fault Price for handling the crisis poorly, failing to go in front of television cameras, explain what was happening and take questions. He read a statement on a press call at 1 a.m. In retrospect, he wishes he had said more that night.

Price left the Iowa Events Center at 2:30 a.m. He picked up a pack of cigarettes and lay down for 90 minutes in the hotel room he had reserved “as a bug-out spot in case things go south.” He showered and walked back to the events complex before sunrise. He worked from there through Thursday before returning to his small office at the party’s headquarters across from the airport. A week later, the one-story, white-brick office was still strewn with boxes. A cake for Mardi Gras lay untouched in the kitchen.

Price said he lost 12 pounds in a week and sought medical care for dehydration. Police were patrolling his street after a sign was found on the interstate that read, “Troy Price will pay the price.”

And he did, announcing his resignation nine days after the meltdown. His successor, Mark Smith, a longtime state representative, was elected Saturday.

Price doesn’t know what he’ll do next, or if he has a future in politics. “This is my home,” he said. “I don’t know where I’d go.”

What he does know, however, is that he’ll be remembered for “a line of code.”

Lenny Bronner, Tony Romm and Neena Satija contributed to this report.