APT has a legion of geeks that help analyze data so companies can make better decisions


Alonzo Benavides, left, and David Kim work on a computer program that will be able to determine the best locations to build businesses during a hackathon at Applied Predictive Technologies in Arlington. (Jahi Chikwendiu/The Washington Post)
October 18, 2013

Step into the offices of Applied Predictive Technologies and you feel like you just entered a playground for young Einsteins.

We’re talking geeks, here.

The first thing you see in the Ballston office’s reception is a Lego model of a Star Wars battlecruiser. Around the corner is a sound-proofed game room where kill-or-be-killed foosball tournaments are held. Cans of WD-40, used to grease the game’s axles, are nearby. PlayStation games play on TV screens hanging on the wall.

The idea here is to keep all these super-noggins happy while they traverse sophisticated mathematics terrain. Their algorithms provide the key to unlocking huge business problems, where billions of dollars are at stake.

The software they create is APT’s firepower as it tracks the clues people leave in their everyday life — from your loyalty card at grocery stores to the check slip you receive at a restaurant to your ATM withdrawals — to help companies make grand decisions out of small samples.

APT’s software tells a convenience store owner whether Snickers or lottery tickets will sell better near the checkout. It tells Victoria’s Secret which lingerie-clad supermodel stimulates the most sales. Wendy’s might ask if it should put smoothies on its summer menu. Choice Hotels wants to know whether its sleek new furniture reels in new customers. Hamburger prices, store hours, wall colors. APT “rapid experimentation” software tests ideas through a business version of a clinical trial, then it tells clients whether it works. The company boasts that $1 of every $5 in U.S. retail transactions runs through its database.

“If I’m a corporation and I have millions to invest, do I invest in advertising? Do I invest in marketing? Do I transform my stores? Do I invest in product selection? Do I train my people differently,” said Rob Palumbo, a managing director at Accel-KKR, a venture capital firm that invested $54 million in APT in 2006. “When you are talking about companies with hundreds of millions, if not billions, in revenues, how you allocate capital means tens of millions of dollars in additional profits.”

APT’s secret sauce has drawn the interest of Goldman Sachs, which recently invested $100 million in the company. That investment alone likely pushed the company’s valuation, a carefully guarded secret, toward several hundred million dollars. If the company goes public, its founders — Jim Manzi, Anthony Bruce and Scott Setrakian — stand to become centi-millionaires.

The investment, one of the largest in the Washington market, drew gasps from the local financial community. Goldman certainly got their attention. With this, big-data analytics deepens its footprint in the region, as APT joins firms such as Clarabridge of Reston, ComScore in Reston, MicroStrategy in Tysons Corner and multinationals like Northrup Grumman and SAIC, which analyze data in the government sector.

“When a Goldman Sachs is willing to put $100 million in your company, you are probably in a very hot space,” said Sid Banerjee, a Washington technologist and chief executive of Clarabridge. “A big check from Goldman also says they have a very optimistic view of your future and confidence in the people running the company.”

That pays for the solutions for some big problems faced by APT’s clients.

“If I’m a CEO of a large retailer and a guy comes in to paint 10 stores blue instead of red, you might think you know the business well enough and can tell whether it helped you make more money or not,” Manzi said. “It turns out that it’s much trickier to reliably identify the cause-and-effect relationships between the change the CEO made and the changes in sales, profits and other outcomes. A million things could have changed. The weather got terrible and forced sales down. Maybe I replaced a sales manager in one store. There is an infinite list of things that are changing and creating effects that are as big as the programs we are executing.”

That explains the stable of math geeks — and Goldman’s $100 million shot of confidence.

“What really stands out for APT is how trusted their results are by their clients,” David Campbell, a vice president in the corporate investing group at Goldman Sachs, said in an e-mail. “For most of the industry it is more of an artform with predictions being highly debatable. APT delivers results that impact the bottom line for its client and, importantly, they are led by a great management team.”

But APT isn’t a slam dunk. An initial public offering, on the one hand, could vault it into the local circle of tech superstars. Or it could end up another AOL, with a fast arc and a quick fall. And just like social media, there’s a lot of competition in big data among companies, all of whom think they have the next big thing.

“There are lots of great ideas floating around Washington about how to leverage data in the security and private sector,”said Nick Nyhan, an expert from the Data Alliance, the advertiser WPP’s strategy group. “Just because you have a lot of data, you can still be wrong.”

An idea springs forth

This all started in 1988 when math whiz Manzi, now 50, told a business consultant about an elaborate test he envisioned for a bank and its entire fleet of branches. His colleague paused, then said simply: “Why don’t you put it in 10 bank branches and see what happens?”

Manzi thought about that solution — for more than a decade.

In 1999, Manzi founded Applied Predictive Technologies with two colleagues, Bruce and Setrakian. Manzi and Setrakian worked at Strategic Planning Associates, a business consulting firm that helps companies improve performance. Bruce was at McKinsey & Co.

“The incident around the bank branches kept bothering me,” said Manzi, who quit high school yet still managed to graduate from Massachusetts Institute of Technology. “I believed this idea of applying rapid, repetitive experiments could be used to make decisions more efficiently. That really was the basic idea behind APT.”

With his savings and some investors’ money, Manzi started the company in his spare bedroom in an apartment overlooking Dupont Circle. Downstairs was a Mrs. Fields Cookies store.

By fall 1999, Setrakian and Bruce joined Manzi in a downtown office. They muddled along until February 2001, when they reached a critical juncture.

“I was very dissatisfied with how things were going,” Manzi recalled. “We were not making sufficient technical progress, not creating work products fast enough.”

Manzi and his team were trying to crack a big problem, which was to create software that allowed companies to design and interpret their own consumer business experiments. Manzi believed he had hit upon the holy grail, which was software that would tell the non-computer-savvy managers whether they should change a hamburger’s price or fine-tune an advertisement for a skirt.

“I got everyone together and said we are going to deliver this thing in late May or we are going to wrap up the company,” he said. “It was crude, held together with baling wire and duct tape, but the software was what I was talking about.”

In September 2001, days after the attacks on the Pentagon and the World Trade Center, they drove to Ohio to pitch their first national retailer, which signed a subscription agreement on a suite of software and paid APT “more money than any of us had ever seen.”

From there, they slowly scaled the company. They became evangelists. They cold-called Fortune 500 firms, auditioning their software like it was the latest video game.

“We really had to go to each one of these companies’ management teams and explain there was a new kind of solution that didn’t previously exist,” Manzi said.

Word of mouth spread. Big names signed on: Wal-Mart, Target, Staples, Intercontinental, Hilton, Starbucks, McDonald’s China.

By 2003, venture capital firms took notice. In 2006, Accel-KKR invested $54 million in APT, buying a majority stake that valued the firm around $100 million. The value is now several times that. APT has offices in San Francisco, London and Taipei, in addition to its Ballston headquarters. The company has more than 100 clients across five continents. It has more than 200 employees, including 150 in Ballston. A Tokyo office is opening soon.

The core of the company is its math geniuses who build the software that allows companies to experiment with business ideas. This allows APT’s customers to more easily factor the dollars and cents implications before strategies are implemented.

In other words, push a button to see why and whether reducing the cost of a six-pack of Coca-Cola by nine cents will bring more total profit, justifying the price reduction. Or test whether the “buy-one, get-one-free” sales for things like sweaters really work if applied to other items already on sale.

“Every year, stores confront this,” Manzi said.

This kind of experimentation isn’t cheap. APT’s multi-year subscriptions for its software starts at several million dollars and rises to double-digit millions over the life of the contract, Manzi said. It offers a suite of six products, known as APT 9. The company’s revenue is in the tens of millions.

And Manzi’s early idea for rapid experimentation? It was pioneered by McLean-based CapitalOne, one of the nation’s largest banks. That makes sense. Its founder and chairman, Richard Fairbank, was the business consultant who suggested Manzi sample branches to test his theory.

The best and brightest

Back at the office, it’s a scorching Friday afternoon and the staff is enduring a weekend hackathon, testing the brain cells of all these number crunchers — known lovingly as “quants.” The food room is packed with organic sweet tea, coconut water, Yuengling beer, Coca-Cola, Fanta and the sort of snacks you’d expect to feed a herd of hyper-metabolic teenagers.

Geoff Patterson, 30, a Princeton computer science major who has been at APT eight years, nibbles from a giant bowl of blackberries as he grinds out a software program that will help him learn more from the billions of bits of transaction data.

A digital clock on a nearby wall signals that there are 18 hours, eight minutes and 32 seconds remaining in the hackathon.

David Kim, 31, a product manager with an econometrics degree from Yale, is playing with a simulation to test the best place to build a $20 million store in downtown Los Angeles.

Just for fun.

“I’m trying to figure out where the customers will come from,” he said. “Whether it will be worth it or whether it will be a waste of money.” Nearby is a lava lamp. Toy guns and Nerf balls rest on the desks. Alonzo Benavides, a physics and computer science graduate of Carnegie Mellon Institute, is clad in his hackathon pajamas.

“Have you nailed down your algorithm yet?” asks one guy, whose sign-on is Newkthewhales. “Because you know we have.”

And these are the geniuses tackling math problems upon which billion-dollar corporate decisions are made?

“The very best people do that extremely well,” said APT co-founder Bruce.

The company’s secret sauce is its computer software, and these folks make it go. They hire the best and brightest minds out of a select group of elite colleges: Dartmouth, Cornell, Harvard, MIT, Princeton, Penn, Virginia, Duke, Maryland, Carnegie Mellon, Georgia Tech, Berkeley and Stanford.

In its search for interns and full-time employees, the company finds people through contacts, job fairs, software events or from other APT employees who are expected to spend 5 percent of their time on recruitment. They take students for coffee, host invitation-only dinners and make personal phone calls to reel in their quarry.

The courtship begins as early as sophomore year in college.

Recruits are wined and dined at top restaurants. After a battery of interviews, the top 3 percent are flown to Washington for a chat. They stay at fancy hotels and tour the city, all on APT’s nickel.

“We pay whatever it takes to get the best people,” Manzi said — beginners earn bonuses and six-figure salaries and veterans several times that. “Our compensation for software engineers is competitive with Google, Facebook and Microsoft.”

Thomas Heath is a local business reporter and columnist, writing about entrepreneurs and various companies big and small in the Washington Metropolitan area. Previously, he wrote about the business of sports for The Post’s sports section for most of a decade.
SECTION: {section=business, subsection=null}!!!
INITIAL commentConfig: {includereply=true, canvas_permalink_id=washpost.com/8bvh5zpd9k, allow_comments=true, commentmaxlength=2000, includeshare=true, display_comments=true, canvas_permalink_app_instance=m6yzjj840m, display_more=true, moderationrequired=false, includefeaturenotification=true, canvas_allcomments_id=washpost.com/km4ey0dajm, comments_period=14, defaultsort=reverseChronological, includevoteofftopic=false, allow_videos=false, includesorts=true, markerdisplay=post_commenter:Post Commenter|staff:Post Writer|top_commenter:Post Forum|top_local:Washingtologist|top_sports:SuperFan|fact_checker:Fact Checker|post_recommended:Post Recommended|world_watcher:World Watcher|cultuer_connoisseur:Culture Connoisseur|weather_watcher:Capital Weather Watcher|post_contributor:Post Contributor, childrenitemsperpage=3, includeheader=true, includeverifiedcommenters=true, defaulttab=all, includerecommend=true, includereport=true, maxitemstop=3, source=washpost.com, allow_photos=false, maxitems=15, display_ugc_photos=false, includepause=true, canvas_allcomments_app_instance=6634zxcgfd, includepermalink=false}!!!

UGC FROM ARTICLE: {allow_comments=true, allow_photos=false, allow_videos=false, comments_period=14, comments_source=washpost.com, default_sort=, default_tab=, display_comments=true, is_ugc_gallery=false, max_items_to_display=15, max_items_to_display_top=3, moderation_required=false, stream_id=}!!!

FINAL commentConfig: {includereply=true, canvas_permalink_id=washpost.com/8bvh5zpd9k, allow_comments=true, commentmaxlength=2000, includeshare=true, display_comments=true, canvas_permalink_app_instance=m6yzjj840m, display_more=true, moderationrequired=false, includefeaturenotification=true, canvas_allcomments_id=washpost.com/km4ey0dajm, comments_period=14, defaultsort=reverseChronological, includevoteofftopic=false, allow_videos=false, includesorts=true, markerdisplay=post_commenter:Post Commenter|staff:Post Writer|top_commenter:Post Forum|top_local:Washingtologist|top_sports:SuperFan|fact_checker:Fact Checker|post_recommended:Post Recommended|world_watcher:World Watcher|cultuer_connoisseur:Culture Connoisseur|weather_watcher:Capital Weather Watcher|post_contributor:Post Contributor, childrenitemsperpage=3, includeheader=true, includeverifiedcommenters=true, defaulttab=all, includerecommend=true, includereport=true, maxitemstop=3, source=washpost.com, allow_photos=false, maxitems=15, display_ugc_photos=false, includepause=true, canvas_allcomments_app_instance=6634zxcgfd, includepermalink=false}!!
Comments
SECTION: {section=business, subsection=null}!!!
INITIAL commentConfig: {includereply=true, canvas_permalink_id=washpost.com/8bvh5zpd9k, allow_comments=true, commentmaxlength=2000, includeshare=true, display_comments=true, canvas_permalink_app_instance=m6yzjj840m, display_more=true, moderationrequired=false, includefeaturenotification=true, canvas_allcomments_id=washpost.com/km4ey0dajm, comments_period=14, defaultsort=reverseChronological, includevoteofftopic=false, allow_videos=false, includesorts=true, markerdisplay=post_commenter:Post Commenter|staff:Post Writer|top_commenter:Post Forum|top_local:Washingtologist|top_sports:SuperFan|fact_checker:Fact Checker|post_recommended:Post Recommended|world_watcher:World Watcher|cultuer_connoisseur:Culture Connoisseur|weather_watcher:Capital Weather Watcher|post_contributor:Post Contributor, childrenitemsperpage=3, includeheader=true, includeverifiedcommenters=true, defaulttab=all, includerecommend=true, includereport=true, maxitemstop=3, source=washpost.com, allow_photos=false, maxitems=15, display_ugc_photos=false, includepause=true, canvas_allcomments_app_instance=6634zxcgfd, includepermalink=false}!!!

UGC FROM ARTICLE: {allow_comments=true, allow_photos=false, allow_videos=false, comments_period=14, comments_source=washpost.com, default_sort=, default_tab=, display_comments=true, is_ugc_gallery=false, max_items_to_display=15, max_items_to_display_top=3, moderation_required=false, stream_id=}!!!

FINAL commentConfig: {includereply=true, canvas_permalink_id=washpost.com/8bvh5zpd9k, allow_comments=true, commentmaxlength=2000, includeshare=true, display_comments=true, canvas_permalink_app_instance=m6yzjj840m, display_more=true, moderationrequired=false, includefeaturenotification=true, canvas_allcomments_id=washpost.com/km4ey0dajm, comments_period=14, defaultsort=reverseChronological, includevoteofftopic=false, allow_videos=false, includesorts=true, markerdisplay=post_commenter:Post Commenter|staff:Post Writer|top_commenter:Post Forum|top_local:Washingtologist|top_sports:SuperFan|fact_checker:Fact Checker|post_recommended:Post Recommended|world_watcher:World Watcher|cultuer_connoisseur:Culture Connoisseur|weather_watcher:Capital Weather Watcher|post_contributor:Post Contributor, childrenitemsperpage=3, includeheader=true, includeverifiedcommenters=true, defaulttab=all, includerecommend=true, includereport=true, maxitemstop=3, source=washpost.com, allow_photos=false, maxitems=15, display_ugc_photos=false, includepause=true, canvas_allcomments_app_instance=6634zxcgfd, includepermalink=false}!!
Show Comments