At Caffe Vivaldi in New York’s Greenwich Village, Peter Muller bangs out a repertoire full of Carole King riffs on the piano along with his own soft-rock compositions that draw on the likes of Van Morrison and Cat Stevens.

“It’s not the same anymore,” he croons. “I’m still looking for my home.”

In the audience, couples sip house cabernet and applaud politely. Some drop $10 tips into a metal bucket.

About 40 blocks uptown, in a 42-story skyscraper overlooking Duffy Square, the straw-haired Muller, 47, performs for a tougher crowd — as the multimillionaire math whiz behind one of Wall Street’s most secretive trading machines.

Muller is the founder of Morgan Stanley’s Process Driven Trading group, or PDT, a 70-person band of PhDs and computer jockeys. They use algorithm-rich programs to bet Morgan Stanley’s money on pricing discrepancies in global markets.

With no outside investors to please, Muller has kept the strategies and performance of PDT under wraps, stoking the curiosity and envy of rivals.

“They say: ‘I know him. He made a boatload of money for Morgan Stanley,’ ” says Arjun Divecha, chairman of Boston-based GMO, who manages about $18 billion using quantitative techniques. “They don’t know how he’s done it.”

Muller makes no apologies for his obsessive secrecy.

“I want my competitors to know absolutely nothing about what we do,” Muller says in his corner office, which is decorated with pictures of his wife, Jillian, and their two children as well as a pair of battered snowboards.

A troubadour, yoga enthusiast and math geek, Muller makes an unlikely Morgan Stanley executive. The 5-foot-10-inch, 160-pound manager practices ashtanga yoga, a style that incorporates synchronized breathing. He’s also a champion Texas Hold ’Em poker player and writes New York Times crossword puzzles several times a year.

Since he started PDT in 1993, its investments have returned an estimated annual average of more than 20 percent through 2010, according to a person close to the group. As a proprietary-trading desk, PDT uses different accounting rules than hedge funds. Its return figure has been adjusted to approximate its performance as if it were a hedge fund.

Hedge funds on average gained 10.4 percent annualized, net of fees, from July 1, 1993, through 2010, according to Chicago-based Hedge Fund Research.

After almost two decades at Morgan Stanley, Muller is about to go out on his own, a move precipitated by the biggest regulatory overhaul of Wall Street since the Great Depression. Banks are jettisoning or closing groups like PDT as a result of the Dodd-Frank Wall Street Reform and Consumer Protection Act, which President Obama signed into law in 2010.

The law’s Volcker rule, named for former Fed chairman Paul Volcker, bars banks from maintaining prop-trading operations and restricts the amount they can invest in hedge and private equity funds to 3 percent of their tier 1 capital. They can also own no more than 3 percent of such funds.

In its 2010 annual report, Goldman Sachs disclosed it had liquidated most of its long-short prop desk positions and was doing the same with its global macro group. Morgan Stanley in March completed the spinning off of FrontPoint Partners, a hedge fund it acquired in 2006. Bank of America disclosed in April a plan to sell its main private-equity business to management.

Now, it’s PDT’s turn. In January, Morgan Stanley said it would spin off the group at the end of 2012 as a separate firm. Morgan Stanley retained an option to buy a preferred stock position in the new company for undisclosed terms.

Cutting risk — and perhaps return

For decades, prop desks have played a crucial role in the transformation of investment banks such as Morgan Stanley from advisory and underwriting businesses to trading powerhouses that bet huge amounts of their own firms’ money. Once the closely held Wall Street partnerships began tapping the public markets for capital, prop desks could wager with shareholders’ money.

By the mid-2000s, prop traders were at the center of the exploding markets for collateralized debt obligations, or pools of bonds, and derivatives, which are instruments that derive their value from an underlying asset. These products were high-octane fuel for the credit bubble that ultimately blew up Bear Stearns, Lehman Brothers and AIG.

As the curtain falls on prop trading at banks, their profits may suffer. Morgan Stanley in 2008 began to pull back on risk, partly by reducing leverage. Since then, its fixed-income trading revenue has been sluggish. Morgan Stanley’s net income fell 45 percent in the first quarter of 2011 from $1.78 billion a year earlier. While firms don’t generally disclose profits from prop trading, Goldman characterized it in 2010 as constituting about 10 percent of revenue in most years.

The loss of this profit engine, along with other Dodd-Frank restrictions on products such as derivatives and debit cards, will hurt Wall Street banks’ returns in coming years, says Charles Peabody, an analyst at Portales Partners.

“It’s the cumulative effect of Dodd-Frank that will lower expected returns,” Peabody says. “Just about every business is facing some kind of crackdown.”

Trick is in the timing

Prop traders aren’t going away; they’re just changing addresses. Dodd-Frank only sought to end the practice in banks to reduce risk. Traders have been relocating to hedge funds and nonbank Wall Street firms such as KKR.

PDT makes most of its money using statistical arbitrage, or stat-arb, former Morgan Stanley employees say. When PDT’s research suggests that a stock is temporarily overpriced, based on its trading history, the group bets against it while piling into corresponding underpriced securities.

In a group of a dozen oil services stocks, six might be rising on a given day and six falling. At some point, the trend reverts: The gainers begin to fall, and vice versa. For PDT, figuring out the precise time to place its bets is the tricky part.

PDT has lost money in only two quarters since its inception and never in a calendar year, according to sources.

Muller skippered PDT through the August 2007 quant meltdown, when overleveraged hedge funds and other investors lost billions of dollars during a four-day period. PDT, along with other Morgan Stanley prop desks, lost $390 million in a single day, according to Morgan Stanley filings. Yet PDT powered back to finish the year in the black, people close to the operation say. The group also lost heavily in the fourth quarter of 2008 amid the global credit crash but still made money for the year.

“I think Peter is brilliant,” says Clifford Asness, co-founder of AQR Capital Management, a $39 billion quantitative investment firm in Greenwich, Conn.

With PDT off to a strong start in the mid-1990s, Vikram Pandit, then Morgan Stanley’s head of institutional equity, wanted to know more about the secretive trading group, says Graham Giller, a researcher who worked with Muller. Giller says Pandit told him to find out details about PDT’s strategies.

“He felt it was the intellectual property of Morgan Stanley,” says Giller, who now runs Giller Investments. Giller says Pandit never set up a meeting at which Giller could inform him of what he had discovered. Pandit, Citigroup’s CEO since 2007, declined to comment.

In going solo, Muller will have to drum up capital from investors who have grown wary of quants, whose returns have lagged those of other types of traditional managers in recent years.

Equity managers using quantitative strategies generated a cumulative return of 37 percent from the start of 2005 through April 2011 compared with 49 percent for equity managers deploying traditional or combined approaches, according to EVestment Alliance, an Atlanta research firm.

As of December, the assets of quant funds were down 10 percent from their peak of $157.5 billion in 2007, as tracked by the research firm Lipper. And new fund launches in 2010 were less than half of the 2005 peak of 189.

“Quantitative managers have been losing market share, and it’s fairly significant,” says Mark Thurston, head of global equity research at Russell Investments.

The underperformance may be due to cyclical forces, as beaten-down value stocks that many quant funds invest in have tended to lag behind their more-expensive growth counterparts in recent years. The decline may also stem from an abundance of quant-managed money pursuing similar strategies.

“Everything goes in cycles,” Muller says. “Success creates more competition.”

The whole and its parts

PDT keeps a low profile within Morgan Stanley’s headquarters. In the elevator vestibule at its ninth-floor offices, a plain, 7-inch-by-7-inch plastic sign reads “Process Driven Trading.”

Muller has created a distinctive corporate culture at PDT, rejecting the eat-what-you-kill ethos practiced at some hedge funds. He cites a childhood book, “Watership Down,” as informing PDT’s environment.

The fantasy novel by Richard Adams tells the story of a nest of rabbits on an odyssey to find a safer home in the English countryside. Each of the principal rabbits excels in an area: intelligence, physical strength, intuition. After tribulations, they make it to their abode.

What does this tale have to do with PDT? “It’s a community coming together with each individual contributing their unique skills,” Muller says. “It’s the whole that benefits.”

PDT has operated as a quasi-independent group within Morgan Stanley for almost 20 years, Muller says. Dede Welles, 41, is the legal head; Amy Wong, 43, serves as operating chief; and Eunice Baek, 41, runs human resources.

“I had grown disillusioned with academia,” says Denis Dancanet, 43, PDT’s head of futures trading, who has a PhD in computer science from Carnegie Mellon University in Pittsburgh. “Maybe three people care what you do.”

Eli Ofek, a New York University professor, also left academia for PDT.

Hard-wired for math

In Manhattan, researchers gather for free-roaming discussions at lunch.

“The range is broad, from the technology to solve Rubik’s Cube to sports to politics to the rate at which flesh-eating bacteria can eat your arm,” says Tushar Shah, 40, PDT’s chief scientist, who has a PhD in physics from MIT.

Muller pays for weeklong vacations for the group to spots such as Grenada and Jamaica to celebrate good years. PDT off-site retreats have included white-water rafting in Maine and paint ball in upstate New York.

Researchers work in teams on PDT’s strategies, with Muller and Shah meeting independently with each group.

Muller and his closest cohorts use a program called an optimizer to allocate assets among the different strategies to generate the most profit with the least risk. Nobody else knows its details.

In the 1990s, Shakil Ahmed, a Yale computer science PhD, worked on U.S. equity stat-arb strategies. Wong, who has an master’s degree in electrical engineering from MIT, oversaw investments based on fundamentals, such as earnings. Giller built a statistical method using interest-rate forecasts. All contributed to PDT’s reputation as a cauldron of white-hot quantitative talent.

Muller says his group will expand after it’s spun off and rechristened PDT Partners. Ownership will be spread among 11 partners. They will pursue strategies that may be less predictable than what Morgan Stanley is comfortable with.

“We have strategies that can take large amounts of capital,” Muller says. “But they are not as consistent. They’ll make money almost every year but not almost every quarter.”

A native of Philadelphia, Muller says he was born hard-wired for math. His Austrian-born father, Kurt, was a chemical engineer with Essex Chemical. His mother, Eva, a native of Brazil, was one of the first women to practice medicine there and later became a psychiatrist.

After graduating from Princeton, Muller took a job as a researcher and programmer at Barra. The pioneering quant firm in Berkeley, Calif., used complex mathematics to help active fund managers measure and control risk.

By 1992, Muller cooked up an idea of using Barra’s quantitative techniques to forecast returns rather than just model risk; Barra could manage money itself. Management turned down Muller’s proposal, and he moved on.

A headhunter put Muller in touch with Morgan Stanley, which was then looking for a quant strategist to drum up business. Muller had bigger aspirations and cut a deal with Derek Bandeen, a prop-trading executive. Muller had two years to get a profitable trading system running. If he failed, he would perform the strategist’s job. PDT was born.

In 1993, as Muller began building a team at Morgan Stanley’s Sixth Avenue headquarters, his business model and ethos mimicked Barra’s, not Wall Street’s.

‘Machine that prints money’

By 1996, PDT was sizzling. “It was like an express train,” marvels Giller, the former analyst. “It was generating millions of dollars a day. It would just appear on the screen. It was surreal.”

In August 2007, Muller confronted the greatest financial storm of his lifetime. Tightening credit markets triggered a panic as leveraged quant funds tried to exit many of the same positions.

“We were all trying to figure out how much selling pressure there was and when it was going to abate,” Muller says.

The PDT founder says he wanted to hold on to his money- losing positions, betting on a reversion. But ultimately Muller agreed with then-CEO John Mack that PDT should reduce its leverage and increase cash. “If we had traded the model without cutting risk, we would have had our best year ever,” Muller says.

The bank and PDT made a similar mistake the very next year, as the subprime contagion morphed into a global rout in the fourth quarter of 2008. Morgan Stanley cut back capital, and the group lost money for the quarter.

In late 2009, Muller says, he approached Gorman, then co-president, to ask for three changes at PDT: Employees should be allowed to directly invest in its funds, PDT should be permitted to employ new strategies to attract outside capital, and employees should own the group in whole or in part.

Gorman, 52, said he could work on addressing the first two requests but resisted the third, sources said.

Today, as regulators write rules for the Dodd-Frank Act, Muller is poised to see his third wish fulfilled. Regulators are set to release a final proposal in October for implementing the ban on prop trading at banks. The Federal Reserve said in February that banks would generally have two years to comply once the rule takes effect.

Quants are laying odds that PDT will prosper on its own. “It’s an incredible opportunity to be spun out,” GMO’s Divecha says. “They have this wonderful machine that prints money. Now, they’ll be using it to print money for themselves rather than Morgan Stanley.”

Burning the ships

As investors still bruised from past losses steer clear of quant funds, Muller is confident that PDT’s reputation will lure money his way. People close to PDT say institutions looking to make investments have already approached Muller.

He will need to contend with critical matters unrelated to trading — accounting, legal, compliance — and build a marketing department.

“Any time a manager goes from a proprietary-trading desk in a large investment bank to an independent business, it can be difficult,” says Robert Frey, who left Morgan Stanley in 1988 to start Kepler Financial Management, a fund he later merged into Renaissance Technologies. “It can be like taking a beautiful hothouse flower and planting it outside.”

Muller co-wrote a song that could be the anthem for his new firm. It’s about the Spanish conquistador Hernan Cortes, who torched most of his fleet upon arrival in Mexico in 1519 to prevent his crew from deserting him on his odyssey of epic plunder.

“The men knew they had to go forward,” Muller sings. “No looking shoreward, forward to glory and gold.”

The full article appears in the August issue of Bloomberg Markets magazine.