CNBC’s Michelle Caruso-Cabrera interviews Stephen A. Schwarzman, The Blackstone Group Chairman at the Institutional Investor Delivering Alpha conference in New York. (Heidi Gutman/CNBC.)

NEW YORK -- Billionaire investor Stephen Schwarzman said Tuesday that President Trump's former business councils crumbled under "astonishing" pressure after violent protests by white supremacists in Charlottesville.

Trump was criticized for wavering on who was to blame for violence at the rallies, prompting days of protests. During a hastily organized conference call with the CEOs on the president's Strategic and Policy Forum, the group's fate quickly became apparent, Schwarzman said at the CNBC Institutional Investor Delivering Alpha conference.

“I asked people what they wanted to do. I gave them one minute each. I wasn’t interested in anyone’s life history,” said Schwarzman, who chaired the group, which also included Jamie Dimon, chief executive of JPMorgan Chase and General Motors' Mary Barra. As the CEOs spoke in alphabetical order the decision became clear, he said.  "By the time you got to W it didn’t much matter."

"People were under legitimate astonishing pressure," he said. "It was pretty clear that the country itself felt like it was was going out of control. We decided there was too much pressure for too many people all running public companies."

Schwarzman, the founder of Blackstone Group,  has emerged as one of Trump's closest allies on Wall Street. His firm is one of the world's largest private-equity and asset management companies. Amid the controversy following Charlotesville, Schwarzman said he received hundreds of hostile emails. "I was accused by people of being a Nazi. I mean, I'm Jewish. It was absurd," he said.

The breakdown of the group was seen as a stinging rebuke to the president, who has made his relationship with the business community a key selling point. Earlier in the conference, Treasury Secretary Steven Mnuchin said he disagreed with the group's decision. "I personally think it was a mistake that the councils were disbanded," he said. "My view is that CEOs being on the councils didn't necessarily mean that they endorsed every single policy of the administration or the president. The purpose of the councils was to give the president advice on issues."

Schwarzman remained largely guarded on his relationship with Trump on Tuesday. They still speak, he said, but when asked how often, he said it was "none of your business."

He also noted that he had worked closely with other presidents, including Barack Obama and George W. Bush during the 2008 financial crisis.

"People who have backgrounds that are relevant" have an "obligation" to help the president and the country when possible, he said. "We have a higher obligation than making money."

Read more: 

When CEOs sound more like the president than the actual president

Trump's business advisory councils disband as CEOs abandon president over Charlottesville reponse