Jay Solomon is not a typical government administrator. In little more than a year since he was appointed by President Carter to run the General Services Administration, the 56-year-old millionaire has developed a reputation for saying almost anything that comes to his mind and for openly admitting the weaknesses of his agency.

Before coming to GSA, Solomon ran the shopping center subsidiary of the giant Arien Realty and Development Corp., which owned or managed 184 shopping centers by the time Solomon left to run GSA.

After graduating from Vanderbilt University in 1942, he helped his father manage a chain of theaters in Chattanooga, Tenn., where he had grown up. When one of their drive-in-theaters was destroyed by a windstorm in 1961, Solomon decided to build a shopping center instead. He then sold the theater chain.

Solomon built other shopping centers and eventually merged his firm with Spartan Industries Inc. in 1971 to form the Arlen company.

Solomon, who is interested in architectural and artistic matters, came to GSA thinking he would beautify federal buildings and attend Cabinet meetings. As the corruption investigations, which started while he was administrator began to gather momentum, Solomon wondered whether he should return to Chattanooga, where he had been happily active in the city's charitable and cultural affairs.

in recent months, however, Solomon became convinced he could not leave what he had started at GSA. He hired Vincent R. Alto, a tough former Justice Department prosecutor, to oversee the investigations and report personally to him. He fired Robert T. Griffin, a veteran GSA official who was second in command, to make it clear to GSA employes that Solomon was in charge.

And he sold all his Arlen stock to avoid any appearance of a conflict of interest in GSA real estate transctions.

As new abuses at GSA have been revealed, Solomon has moved to correct them. He ordered stockpiles sold with competitive bids. He stopped purchases of high-priced office machines. He reinstated "whistleblowers" who had been fired by GSA for disclosing abuses. He insisted on separate inspections of repair and maintenance work. And he replaced officials who had responsibility over corrupt departments.

Federal authorities involved in trying to help reform GSA believe some of Solomon's actions have been superficial responses to deep problems and wonder if he, or anyone, can take the drastic steps necessary to change the system.

But they welcome Solomon's openess and believe it is the most effective contribution he can make at this point. "It's a refreshing change," said once Justice Department official, "from previous GSA administrators."