Wall Street's top regulator went to the heart of Silicon Valley this week to defend the public markets and warn tech startups of the perils of staying private too long.

Mary Jo White, chair of the Securities and Exchange Commission, took particular note of the world's 150 so-called unicorns, tech companies valued at $1 billion or more, but still in private hands.  "Beyond the hype and the headlines, our collective challenge is to look past the eye-popping valuations and carefully examine the implications of this trend for investors," White said in a speech at Stanford Law School late Thursday.

"These are areas of concern for the SEC."

White did not mention any specific companies, but investors are patiently waiting for Uber, Airbnb and Snapchat to all go public. They are worth $62 billion, $25.5 billion and $16 billion respectively, according to Fortune.

They are among a growing group of Silicon Valley tech firms that are eschewing tradition and staying in private hands, rather than joining their brethren in the public markets where mom and pop investors could buy shares in the company. By stubbornly staying on the sidelines these companies are helping suppress the already dismal initial public offering market this year, critics say. And now they're starting to catch the attention of the country's chief financial regulator.

"Silicon Valley has become home to many private companies with valuations that exceed many listed ones that in another time would be publicly traded," White said.

White echoed the concerns of some industry insiders that these tech start-ups are missing out on the market discipline public companies receive by being accountable to the whims of public shareholder. For these companies to be successful, investors must be confident that they are being treated fairly and that companies are being transparent, she said.

"As the latest batch of start-ups mature, generate revenue, achieve significant valuations, but stay private, it is important to assess whether they are likewise maturing their governance structures and internal control environments to match their size and market impact," said White.

If and when these companies do decide to public, they could provide a needed boost to the market. There have been only a handful of small IPOs so far this year amid market volatility. It is the slowest start to the year since the year since the 2008 financial crisis, according to according to Renaissance Capital, a manager of IPO-focused exchange traded funds.

But it may be an uphill battle.

Recently pressed on when ride-hailing company Uber would go public, Travis Kalanick, the company's chief executive, balked.

"I'm going to make sure it happens as late as possible," Kalanick said on CNBC. Perhaps in a year or two, he was asked. Five? "I mean I'll, I'll keep you posted, I have no idea," he said.

Valued at $62 billion, Uber is worth more than Ford or Macy's. It could soon approach the market value of defense contractor Lockheed Martin, which makes the F-16 fighter jet.

"We've raised...[in] the last 18 months something like, somewhere in the neck of the woods of $10 billion,"Kalanick said. "We're not in need of public capital."

Read More: