Two decades ago the American Enterprise Institute, a think tank whose newest scholar is former President Ford, was an $80,000-a-year right-wing propaganda mill operating out of a hole in-the-wall on F Street NW.

Today its well-appointed offices are spread over four floorsof a modern building and its annual budget is $5 million. Even more important, it has the prestige to draw on the talents of a former President of the United States.

The man who made the difference is William J. Baroody, 61, a sloe-eyed Lebanese American apostle of free enterprise, who said in a recent interview that he was a "relatively brash young man" when he left the security of the U.S. Chamber of Commerce to a dismal future at the time I took it over."

He began changing AEI's image "by dumping the business types" from the institute's academic advisory board.

He replaced them with conservative but respected economists such as Milton Friedman, from the University of Chicago, who won the 1976 Nobel Prize in economics, and Paul McCracken and Herbeert Stein, both of whom became chairmen of the Council of Economic Advisers under President Nixon.

By 1964 AEI was housing numerous members of candidate Barry Goldwater's presidential brain-trust, including the Rev. Stanley J. Parry, a Notre Dame political scientist who was one of Goldwater's academic advisers. J. William Middenforf II, who ran Goldwater's finance committee, and Peter O'Donnell, who headed the draft Goldwater committee. Middendorf and O'Donnell were AEI trustees.

The institution had clearly become more respectable, but it was still closely identified with the right wing of the Republican Party. One problem that had to be overcome, according to Baroody> was the notion that AEI was a captive of business.

He had already gotten rid of the businessmen on the academic advisory board. The next step was to replace some of the business money that kept AEI alive with foundation money. Today a substantial portion of AEI's budget comes from foundation grants.

Yet until very recently, the image problem still seemed to bother many AEI scholars.

People at AEI are quick to mention that they have received a research grant from the Ford Foundation, a badge of instant respectability.

But much of AEI's $5 million annaul budget still comes from coroporations, unlike the Brookings INstitution, which relies on its $35 million endowment and foundation support for almost all of its $7 million budget.

And AEI's board of trustees consists entirely of executives from such corporate giants as Mobil Oil, Standard Oil of California, Continental Can, Procter & Gamble, Rockwell International and Libbey-Owens-Ford.

The institute also carries on a major non-research activity aimed at least in part at image-building. Its videotaped forums and debates featuring both liberal and conserative policymakers and scholars are shown on hundreds of local television stations across the country.

"The forums are, frankly a means of gaining respectability," said Robert J. Pranger, who heads AEI's foreign and defense studies program.

AEI has also begun to attract a new breed of scholar in the last year.

Irving Kristol, social philosopher, editor of the Public Interest, former solcialist, liberal and now self-designated "neo-conserative," has joined up.

Ben J.Wattenberg, a former speech-writer for President Johnson and an outspoken member of the center-right Coalition for a Democratic Party, will be spending about half this time at AEI.

Austin Ranney, a professor of policital science long identified wiht liberal Sen. Hubert H. Humphrey, is a resident scholar at the institute.

Baroody denies operating "an affirmative action program" to recruit Democrats but, especially in the areas of foreign and defense policy, he said, "we're drawing from both parties."

But the tone remains clearly conservative. Other new appointments at AEI include former Solicitor General Robert H. Bork, who carried out President Nixon's instructions to fire special prosecutor Archibald Cox after Attorney General Elliot L. Richardson and Deputy Attorney General William D. Ruckelshaus residned rathe than do so.

Before leaving office, Treasury Secretary William E. Simon indicated that AEI had a key role to play in helping the Republican Paryt rebuild for 1980.

"We ought to take 50 or 100 issues and get an intellectual response to those questions," Simon said. The Democrats "have Brookings and we ought to use AEI - and we will over the next four years."

AEI has convinced Rep. Henry S. Reuss (D-Wis.), who chairs the House Banking, Currency and Housing Committee, that it has earned the right to be taken seriously. "I think its intellectual standards are high," Reuss said.

Sen. William Proxmire (D-Wis.), a member of the Appropriations Committee and the Joint Economic Committee, said AEI's studies on economic policy "compare quite well" with those published by Brookings.

AEI also studies foreign and defense policy, government regulation and government. And it publishes legislative analyses and a daily news summary, mainly for legislators.

According to Baroody, the main difference the change of administrations is likely to make for AEI, except for the addiiton of Ford, is that the institute will be called upon by the Carter administration to provide a critical perspective.

The Nixon and Ford administrations did not use AEI the way Democratic administrations have grown accustomed to using Brookings - for aid, comfort and advice. "Somehow, Henry Kissinger never felt the need to call on me," said Pranger, the foreign affairs specialist.

But if Simon and Baroody are both right, AEI may now find its services in demand as never before.