“It’s an example of what we can do well when we put our resources on an enterprise project and we will continue to do so,” said the top editor, according to a source.
Opinions vary as to just what sort of example “The Big Hack” presents. The story certainly packed a set of dramatic allegations, namely, that operatives with the Chinese People’s Liberation Army had planted chips in server motherboards manufactured in China by subcontractors for San Jose-based Super Micro Computer (or Supermicro). “In Supermicro, China’s spies appear to have found a perfect conduit for what U.S. officials now describe as the most significant supply chain attack known to have been carried out against American companies,” reads the investigation by Jordan Robertson and Michael Riley.
“Most significant” is an understatement. This was a “WHOA, if true” sort of story. For years, U.S. government officials and industry bigwigs have fretted over the threat of a backdoor in the supply chain. There’s no telling whose computers could be compromised, including those of key U.S. agencies. Worries of this type explain why the U.S. intelligence community has warned against using the products of Chinese tech giants Huawei and ZTE. Allowing companies with close ties to foreign governments to crack the domestic telecom market, said FBI Director Christopher A. Wray in February, could pave the way to information theft and “undetected espionage.”
The Bloomberg investigation signaled that this nightmare could already be unfolding. Apple, according to three “senior insiders” cited by Bloomberg, found “malicious” chips on their Supermicro motherboards. Amazon, too, appears in the story: When the retail giant (whose chairman and CEO, Jeffrey P. Bezos, owns The Washington Post) was checking out a video firm called Elemental Technologies for a possible acquisition, it commissioned a security review that covered the company’s Supermicro servers, according to Bloomberg. “Nested on the servers’ motherboards, the testers found a tiny microchip, not much bigger than a grain of rice, that wasn’t part of the boards’ original design. Amazon reported the discovery to U.S. authorities, sending a shudder through the intelligence community,” writes Bloomberg.
In all, reports Bloomberg, the hack affected nearly 30 companies. Or maybe not — denials piled on top of the reporting. Here you go:
“It’s untrue that AWS knew about a supply chain compromise, an issue with malicious chips, or hardware modifications when acquiring Elemental,” Amazon wrote. “On this we can be very clear: Apple has never found malicious chips, ‘hardware manipulations’ or vulnerabilities purposely planted in any server,” Apple wrote. “We remain unaware of any such investigation,” wrote a spokesman for Supermicro, Perry Hayes.
Following the story’s publication, other voices belted out their skepticism. “The Department of Homeland Security is aware of the media reports of a technology supply chain compromise. Like our partners in the UK, the National Cyber Security Centre, at this time we have no reason to doubt the statements from the companies named in the story,” said a statement from DHS. On Thursday, Director of National Intelligence Daniel Coats told Cyberscoop that he’d seen no evidence of Chinese hacking into Supermicro motherboards.
And in a story covered by Bloomberg itself, a key National Security Agency (NSA) official at an Oct. 10 appearance made some unusual remarks about the story. Rob Joyce, a cybersecurity official, sounded almost like a journalist seeking a scoop when he said, “If somebody has first-degree knowledge, can hand us a board, can point to somebody in a company that was involved in this, as claimed, we want to talk to them,” Joyce said.
Why would Joyce be making an appeal for such evidence? Because the Bloomberg story didn’t furnish much of it. The lack of goods surfaced on the story’s presentation on the cover of Bloomberg Businessweek.
Reasonable readers would be forgiven for concluding that the tiny thing on top of the finger is a malicious chip threatening the U.S. economy. To its credit, the magazine’s inside included an explanation that “Microchips found on altered motherboards in some cases looked like signal conditioning couplers.” To its discredit, the magazine’s inside included an explanation that many people likely missed. A Bloomberg spokesperson said that the chip is “described in the magazine and on the web site. Nowhere in the story does it say it is the actual chip.”
That Bloomberg would get its hands on a specimen of international hardware hacking may well be an extravagant expectation. Yet the Bloomberg story also whiffs on lower-level evidence thresholds. For the most part, the allegations rest on human sources: “In all, 17 people confirmed the manipulation of Supermicro’s hardware and other elements of the attacks. The sources were granted anonymity because of the sensitive, and in some cases classified, nature of the information,” reads the piece.
Anonymous sources can be very good sources. They’re the serial stars of Bob Woodward’s investigative nonfiction; they’ve played a central role in career-killing #MeToo pieces; and they undergird nearly every major national-security scoop churned out by major news outlets. At first blush, Bloomberg had a surfeit of such sources: 17 people attesting to a wide-ranging hardware hack.
Read the story, however. This particular hack spans the Chinese People’s Liberation Army, Supermicro, Amazon and its Amazon Web Serivces, Apple, the FBI, the Defense Department, the White House and unnamed companies. For a more detailed picture of this story’s sprawl, consider that it alleges that Apple alone:
- Planned to order in excess of 30,000 Supermicro servers for a global expansion;
- Found malicious chips on Supermicro servers in 2015;
- Espied “odd network activity and firmware problems,” triggering the discovery of the chips;
- Reported the discovery to the FBI but kept the matter secret internally;
- Arranged a closer relationship with Supermicro after 2013;
- Launched a project known as “Ledbelly” aimed at making faster the search function for Siri;
- Planned in 2014 to “order more than 6,000 Supermicro servers for installation in 17 locations, including Amsterdam, Chicago, Hong Kong, Los Angeles, New York, San Jose, Singapore and Tokyo, plus 4,000 servers for its existing North Carolina and Oregon data centers. Those orders were supposed to double, to 20,000, by 2015”;
- Didn’t allow government investigators to access its facilities;
- Removed all of its Supermicro servers within weeks of finding the hack.
Seventeen sources suddenly doesn’t feel quite so luxurious. Here’s one of the key assertions backed up by a single source: “Apple made its discovery of suspicious chips inside Supermicro servers around May 2015, after detecting odd network activity and firmware problems, according to a person familiar with the timeline.”
In an interview with the Erik Wemple Blog, George Stathakopoulos, Apple’s vice president of corporate information security, provided a heated rebuttal of the Bloomberg allegations against his company. No, the company never found a hardware implant; no, the company never removed its Supermicro servers and many are still in place; no, the company never communicated with law-enforcement officials about this alleged malicious hardware implant, says Stathakopoulos.
“Not only haven’t we found anything but we have never seen a shred of evidence” regarding the malicious chip nor any indication of “what the chip looks like,” Stathakopoulos says.
What Apple did do, according to the security executive, was rummage. After receiving the allegations from Bloomberg, Apple reopened its files, interviewed all relevant personnel, looked at procurement receipts and so on. “We did deep investigations into the hardware,” Stathakopoulos says. “Each time we were contacted by reporters, I was able to walk them through all the things we did.” Extensive internal investigations must always precede wide-ranging denials, and these exercises have occurred before in response to Bloomberg tech reporting. In 2014, Bloomberg’s Riley reported that NSA allegedly exploited the so-called Heartbleed bug for intelligence, though the agency denied the finding. “When we are going to make a flat out denial, we have a responsibility to make sure that we are 100% certain of the facts. While we were busy ensuring that no one in the agency was aware of the Heartbleed vulnerability, Bloomberg published the article with the false claim,” a spokesman for the director of national intelligence told the Erik Wemple Blog at the time.
And each time Apple was contacted by the Bloomberg reporters, claims a company insider, the allegations shifted in magnitude. In the first go-round, in October 2017, the Bloomberg reporters alleged that there were “hundreds” of servers that had carried the malicious chips; then, in June 2018, the number had dwindled to “multiple” compromised servers; in the final story, there was even less specificity: Servers were allegedly found to be compromised by Apple in May 2015.
Apple blasted the Bloomberg story on more than one front, issuing statements to the media and also to Congress. In a letter to two committees, Stathakopoulos wrote, “In light of your important leadership roles in Congress, we want to assure you that a recent report in Bloomberg Businessweek alleging the compromise of our servers is not true.” In its initial pushback, the company relied on denials of this sort rather than requesting a retraction, out of concern that too forceful a response would be viewed as an abuse of power.
On Friday, however, Apple CEO Tim Cook tossed out the caution, demanding that the story be retracted. “There is no truth in their story about Apple,” Cook told BuzzFeed News in the company’s first public demand of this nature. “They need to do the right thing and retract it.” Since the first objections arose, Bloomberg has stuck with a single statement vouching for the work of its editorial staff: “Bloomberg Businessweek’s investigation is the result of more than a year of reporting, during which we conducted more than 100 interviews. Seventeen individual sources, including government officials and insiders at the companies, confirmed the manipulation of hardware and other elements of the attacks. We also published three companies’ full statements, as well as a statement from China’s Ministry of Foreign Affairs. We stand by our story and are confident in our reporting and sources.”
According to a company source, editorial staff has been “frustrated” that competing news organizations haven’t managed to match the scoop. Sources tell the Erik Wemple Blog that the New York Times, the Wall Street Journal and The Post have each sunk resources into confirming the story, only to come up empty-handed. (The Post did run a story summarizing Bloomberg’s findings, along with various denials and official skepticism.) It behooves such outlets to dispatch entire teams to search for corroboration: If, indeed, it’s true that China has embarked on this sort of attack, there will be a long tail of implications. No self-respecting news organization will want to be left out of those stories. “Unlike software, hardware leaves behind a good trail of evidence. If somebody decides to go down that path, it means that they don’t care about the consequences,” Stathakopoulos says.
In the face of challenges to the story’s veracity, Bloomberg has commissioned additional reporting to reinforce its initial findings. One of the story’s reporters, for example, contacted a former Apple employee on Oct. 10 seeking information on the alleged purge of Supermicro servers, according to correspondence reviewed by the Erik Wemple Blog. We asked Bloomberg about any additional reporting on the alleged hack. “We do not comment on our unpublished newsgathering, editorial processes, or plans for future reporting,” replied a company spokeswoman.
As for Amazon, according to the Bloomberg story, the company reacted quickly to the alleged threat from the implanted chips. After it allegedly got word of the bad stuff on Elemental’s servers, for example, Amazon alerted authorities, “sending a shudder through the intelligence community. Elemental’s servers could be found in Department of Defense data centers, the CIA’s drone operations, and the onboard networks of Navy warships. And Elemental was just one of hundreds of Supermicro customers,” notes the story. Yet Amazon isn’t copping to such foresight. Instead, Amazon Web Services CEO Andy Jassy joined the call for a retraction:
Amazon, Apple, the FBI and the Defense Department are all large bureaucracies that tend to create records of transactions, meetings, controversies, catastrophes. A Chinese hardware hack would surely have generated thousands of such records in the form of emails, text messages and the like. Asked whether the Bloomberg reporters have vacuumed up any such material, the Bloomberg spokesperson replied, “We do not comment on our unpublished newsgathering, editorial processes, or plans for future reporting.”
That no-comment “response” served as a global Bloomberg response to more than 15 questions posed by the Erik Wemple Blog.
The best journalism lends itself to reverse engineering. Though no news organization may ever match the recent New York Times investigation of Trump family finances, for instance, the newspaper published documents, cited sources and described entities with a public footprint. “Fear,” the recent book on the dysfunction of the Trump White House, starts with the story of a top official removing a trade document from the president’s desk, an account supported by an image of the purloined paper.
Bloomberg, on the other hand, gives readers virtually no road map for reproducing its scoop, which helps to explain why competitors have whiffed in their efforts to corroborate it. The relentlessness of the denials and doubts from companies and government officials obligate Bloomberg to add the sort of proof that will make believers of its skeptics. Assign more reporters to the story, re-interview sources, ask for photos and emails. Should it fail in this effort, it’ll need to retract the entire thing.
There’s just too much at stake here. Supermicro’s stock, for starters, took an Acapulco dive following publication of the Bloomberg investigation. It hasn’t much recovered, denials notwithstanding. The company tells the Erik Wemple Blog that it “only became aware of the specifics of these allegations when the article was published.”
So, Bloomberg has some options, none of which is standing pat and hoping that the next Trump scandal distracts the body politic. Your move, Bloomberg.
More reading on this story: BuzzFeed’s story on Tim Cook’s retraction request provides a comprehensive look at the issue; Motherboard did an evaluation that covered a lot of shoreline; Lightbluetouchpaper.org explains that the attack outlined in the Bloomberg story passes the “sniff test”; Business Insider chronicled doubts among security experts; Axios characterized the story as “fraying”; ZDNet reported on the unsettling remarks of a security expert quoted in the Bloomberg story.