The American Enterprise Institute, a Republican think tank that helped fill the ranks of the Reagan administration in its first term, is facing financial difficulties that have forced across-the-board staff cutbacks and other reductions, according to AEI members.

AEI President William J. Baroody Jr. two weeks ago informed staff and scholars at the 42-year-old institute that staff would be cut immediately by 15 to 20 percent, publications scaled back and a planned move to newer, posher quarters abandoned because of financial troubles.

Baroody said yesterday the difficulties are caused primarily by foundation and corporate grants to AEI not keeping pace with the rising expenses of running the institute.

After growing "like Topsy," as one AEI official put it, in the last 10 years, AEI is being forced to reduce its budget from about $12.7 million this year to $12.5 million next year.

Unlike the Brookings Institution, the generally Democratic think tank which has a hefty endowment, AEI must raise its entire budget every year, primarily through private grants, Baroody said.

Competition for these sources of income has gotten much more intense in the last few years as cutbacks in federal programs have forced a new array of nonprofit groups into the race for private dollars, he said.

Others familiar with AEI's financial difficulties said that at the same time, AEI, which long ago carved out a niche as a conservative Republican think tank, increasingly has come under pressure from the Heritage Foundation, which aggressively advocates specific conservative positions on whatever may be the issue of the day.

Republican activists said this week that as a result some right-wing funding is now going to Heritage instead of AEI, which is more academic and does not take positions as an institution.

Those inside and outside AEI said also that some of the money troubles are due to the lack of long-range financial planning. "Things weren't managed properly, budgeted properly," one AEI scholar said.

"There wasn't a master plan," another added. For instance, an effort by former AEI head William J. Baroody Sr. to set up a permanent endowment was not pursued after he died.

Several AEI officials said this week that it is unclear what effect the cutbacks will have on the institute, which sent 25 to 30 top people into the Reagan administration during its first term.

Among them were Jeane J. Kirkpatrick, former chief delegate to the United Nations; Arthur Burns, former ambassador to Germany; James C. Miller III, director of the Office of Management and Budget, and David Gergen, former communications director at the White House.

Baroody said that no senior fellows have been cut from the institute, although some "adjunct" scholars around the country who receive funding from AEI have seen their grants cut back or eliminated.

But staff cuts in other areas -- some 20 to 30 people were given pink slips -- means that some scholars for the first time may have to share research assistants.

While the number of publications put out under the AEI label will be reduced, officials said, it is unclear whether some of them, such as Regulation magazine, will be folded. Another magazine, Public Opinion, is looking for private investors.

Norman J. Ornstein, a congressional expert who has been a fulltime scholar at AEI for the last two years, said this week it "remains to be seen" how the belt-tightening will affect AEI.

He said he thought it could be beneficial in the long run, providing "an opportunity to rethink the mission and streamline."

Another scholar added, "There's sort of a grim mood, but a lot of us feel it can be turned around."