As organizations and industries realize the benefits of using AI, from wildlife conservation to health care, they are grappling with a common but urgent need: how to store and process massive amounts of data.

In London, this challenge involves saving a treasured national icon: the hedgehog. They’ve always occupied a special place in Britons’ hearts¹, winning a national poll as Britain’s favorite animal, but the number of these adorable creatures that fit in the cup of your hand² has shrunk by 30 to 75 percent since 2000 due to habitat fragmentation and loss and they’re vulnerable to extinction.

To research and reverse this decline, the Zoological Society of London (ZSL) created the London HogWatch³ project to analyze over 15 million images captured by 4,000 camera traps throughout greater London. ZSL is using an advanced AI solution designed by PEAK:AIO, a company that develops storage solutions for AI workloads, and powered by high-density storage from Solidigm, a leading global provider of NAND flash memory solutions. Solutions like these are necessary because AI requires a multitude of steps and data.

AI image analysis is also at the core of revolutionary advancements in other industries, such as agriculture and health care. Farmers in the United States are using drones that can take 20 snapshots per second of crop fields. Those images are then compared by AI to a database containing more than a million images to determine which specific areas in a field need management. This approach enables more precise and effective weed control while reducing the amount of herbicide required and costs. Meanwhile, doctors are using AI-reviewed MRIs, X-rays and other diagnostic tools to quickly and accurately diagnose illnesses and tumors⁴, reducing patients’ scan times by half. The faster, more accurate diagnosis lets health care providers start appropriate treatments sooner, improving patient outcomes⁵.

AI innovations can solve problems once considered insurmountable, but they’re running headlong into multiple barriers such as storage capacity, performance, physical footprint and power constraints. Training an AI system — whether it’s a large language model (LLM), a large video model (LVM) or a large image model (LIM) — has a mind-boggling amount of data that needs to be stored, retrieved, processed and analyzed. For example, the cameras farmers are using to photograph their fields require six terabytes per day of storage (roughly equivalent to 3,000 hours of HD video streaming) per sprayer, while the London HogWatch project intakes millions of wildlife images that need constant, high-speed access.

“The generative AI models that are dealing with richer forms of media like images and video might train on petabytes or tens of petabytes of data,” said Ace Stryker, Solidigm’s director of market development. (One petabyte equals 1,000 terabytes of data.)

That’s why new data storage infrastructures utilizing SSDs are essential for the growth of AI models. Modern SSDs are highly scalable, making it possible to store 2PBs of data in a single 1U chassis. For example, the Solidigm D5- P5336 QLC SSD currently offers more than 61TB of storage in a single device about the size of a deck of cards, and soon that will double with up to 4PBs in a 1U rack based on their  122TB drive launching in early 2025⁶. Because they can be placed right at the edge, closer to the source of the data, they also enable the flexibility today’s ambitious AI applications require. 

With high-density SSDs like these, the more data put into a single drive, the fewer drives a company will need, saving both power consumption and rack space in the data center. According to Stryker, if an AI cluster is run on old storage infrastructure like hard drives, a company might be allocating as much as a third of their power on storage. “By making smart storage choices, you can improve both performance and power efficiency,” said Stryker. By contrast, the densest mechanical hard drive currently tops out at about 30TB in a significantly bigger form factor.

Ocient, a hyperscale data analytics solutions provider, is one company leading the charge in making AI data centers more efficient with high-density SSDs⁷. “The energy consumption required for compute-intensive AI/ML workloads has reached unsustainable levels and, as a result, has repositioned the foundations of this critical technology — computing power and data storage — as essential enablers for future innovation and growth,” explained Chris Gladwin, CEO of Ocient. “Hardware like Solidigm’s high-capacity SSDs combined with software solutions capable of delivering maximum efficiencies on energy consumption, storage and data performance will be essential for driving sustainable innovation across industries.”

AI workloads have specific storage requirements at each stage. High write performance is crucial during initial data ingestion, while the training phase requires fast random read capabilities to keep expensive GPUs working at full capacity. “If you’re pairing those with slow storage, then you have a GPU utilization problem,” noted Stryker. “You’ve spent a lot of money on high-power compute parts that aren’t as busy as they could be because they’re not being fed data fast enough.”

When the London HogWatch team saw hedgehog images coming in from their camera traps, they were excited about gaining detailed insight into where hedgehogs lived and how they traveled across the city. However, their traditional storage solutions couldn’t process and store millions of these images within their power constraints.

“Traditional flash-based storage couldn’t deliver the required 3PB capacity within the available power limits, nor could it provide the data transfer speeds necessary for continuous GPU operation,” said Mark Klarzynski, CEO of PEAK:AIO. They addressed this challenge by designing a solution built around Solidigm’s high-density D5-P5336. “The D5-P5336 allows ZSL to avoid compromises on dataset size or performance, both of which are critical for the success and scope of their conservation projects .”

The conservationists at ZSL are now in a better position to determine how many hedgehogs currently call London home and they can identify the green corridors that hedgehogs use to travel across the city. Such mobility is crucial for ensuring that these animals retain the genetic diversity that will allow their population to rebound in the future.

“The emergence of big data in ecology has revolutionized data collection, but it has also created challenges,” said Chris Carbone, professor in macroecology and conservation at ZSL. “High-capacity storage solutions, like the newly acquired supercomputer system, are allowing us to efficiently store, manage and analyze terabytes of data and has transformed conservation work at ZSL.”

Beyond the London HogWatch, ZSL is now able to leverage AI and data storage for climate change monitoring, develop global biodiversity metrics through data mining, and use underwater cameras and drones for marine conservation. As Carbone and his colleagues have proved, the right storage infrastructure turns AI into a powerful and essential tool, driving powerful innovations that once seemed impossible.

“The ability of SSDs to scale in capacity is a lot greater than what we think hard drives will be able to achieve in the future,” Stryker said. “That’s a trend that we expect to continue.”