Former deputy defense secretary Frank C. Carlucci left the Pentagon five years ago when its budgets were growing rapidly and the military was filled with an optimism engendered by the greatest rearmament in peacetime history.

Now, as President Reagan's nominee to succeed Defense Secretary Caspar W. Weinberger, Carlucci will inherit a Pentagon that is approaching the bleakest 14 months of the eight-year Reagan administration.

He will return to a somber Defense Department, worried about how to scale back a buildup it can no longer afford and how to maintain the brightest, best-educated volunteer military in history when the pool of youths is declining and departures are increasing among its most specialized officers.

He will have to combat those problems at the same time that the end of the Reagan administration is impelling more high- and mid-level Defense Department officials to leave. Replacements for them are not only becoming harder to find, according to Pentagon officials, but a sluggish confirmation process in the Senate delays those who have been found from taking over empty jobs.

Carlucci also assumes control of a Persian Gulf operation that becomes more controversial and costly by the week. Although Carlucci, as chief of the National Security Council, helped orchestrate the plan to provide U.S. military escorts for reflagged Kuwaiti tankers, it has evolved into a military operation guided primarily by the defense secretary and the Joint Chiefs of Staff and the policies they set in reaction to day-to-day events.

Perhaps the one bright spot in his 14-month tenure will be the opportunity to give the Pentagon a different role in the arms control debate, in which his predecessor has been a vocal negative influence. Carlucci brings a more moderate voice that could put the Pentagon more in line with the prevailing views of a Reagan administration bent on signing arms-control accords and could soothe the political strains that have grown between the State and Defense departments in recent years.

"Clearly there are severe limits on what anyone can do in the final year of any administration," said House Armed Services Committee Chairman Les Aspin (D-Wis.) in looking ahead to the Carlucci months. "Mr. Carlucci can, however, have a major impact on the two core policy issues staring us in the face -- the Persian Gulf and arms control. The final judgment on the next defense secretary is likely to rest with his conduct on those issues."

Although those two issues may receive more public attention than any other during his tenure, Carlucci's most serious problem will be how to deal with a defense modernization program that was created when budgets were wildly generous but now will grow little, if at all.

Some programs will have to shrink or die.

Carlucci can either begin cutting back in the next 14 months, or leave it to the next administration. Most Pentagon analysts believe the hard choices should be made sooner rather than later.

"I am more worried about leaving the budget situation the way it is than anything else," Weinberger said at a Pentagon news briefing yesterday. "For the next couple of years, there's a lot left in the pipeline that will be delivered. But you never worry about tomorrow necessarily in this job -- you always worry about . . . two, three five years ahead."

Even without the pending cuts mandated by the Gramm-Rudman-Hollings balanced-budget law, internal Pentagon reports have warned that the military cannot afford to execute much of the buildup already planned. One report estimated that even with regular budget increases, the Pentagon will fall $2 trillion short of what is needed for the buildup as now planned by the turn of the century.

Decisions on how to cope with a diminished budget will have to be made in coming months. In many cases, Pentagon officials must decide soon whether it is in the best interests of national security and diminishing budgets to drop some programs, or merely reduce the numbers of items and stretch out the time taken to put them into service.

Among the large number of decisions on individual weapons systems the Pentagon must determine are whether the Army should be given the new LHX helicopter program and how many new Bradley Fighting Vehicles it should get.

Also needed are new strategies for financing and manning the Navy's 600-ship fleet in an age of declining manpower and funds for operating the vessels already in service or under construction.

Controversial decisions remain to be made on correcting problems with some of Reagan's most cherished yet troubled programs, among others the B1 bomber and the MX missile.

At a time when money is becoming scarcer, the Pentagon is still confronting issues of fraud, waste and abuse in its procurement system, issues that Carlucci was battling during his service as Weinberger's deputy in 1981 and 1982.

Although it is an issue Carlucci may be able to ignore, the Pentagon is getting the first hints that its much-praised efforts to improve the quality of the armed forces will be hard to maintain. The pool of eligible young males is dwindling, which some Pentagon officials say will require a major realignment of jobs open to women in order to keep billets filled.

In addition, while recruiting offices have enjoyed a record success rate in recent years, Pentagon officials have detected the first signs that goals may be increasingly harder to meet. For the first time in several years, some offices have not met recruiting targets and the Army has been rocked by scandals at some offices where recruiters have violated testing and screening rules to meet quotas.