Five years into the biggest peacetime military buildup in history, Defense Secretary Caspar W. Weinberger and his defense budget appear to have worn out their welcome in Congress.

Interviews across the political spectrum show that Weinberger has lost both credibility and clout this year and that President Reagan's military spending wave has crested.

Sens. David H. Pryor (D-Ark.) and Lawton Chiles (D-Fla.) said that if they were advising Reagan they would recommend that Weinberger resign, so the Defense Department could regain credibility. Other senators said the same thing privately but asked not to be quoted by name.

Weinberger, in an interview, said he has no intention of quitting because he has "a lot more to do." He brushed aside criticism, saying: "It's like being a judge in a beauty contest. The longer you do it, the more people you disappoint."

But as the Senate prepares today to resume debate on the defense authorization bill, it is clear that attitudes about the Pentagon budget and its chief advocate have changed dramatically.

Pryor went back to his successful election campaign to explain why he thinks that it has happened.

"I was sitting at a picnic table eating catfish with a bunch of retired people in northern Arkansas," he said, "when this old gent came up to me, threw a newspaper clipping down at my place, and asked: 'Now what are you going to do about this?' "

The clipping was about the $7,400 coffee maker the Air Force had bought for its C5A cargo plane, Pryor said -- one of dozens of Pentagon horror stories he was confronted with as he sought the votes of the traditionally pro-defense people in his state.

"They wanted us to spend more for defense," Pryor said, "but they didn't want their money wasted. They talked about graft. They were incensed, frustrated, angry. I talked to Barry Goldwater about it. He said, 'I know. I hear the same thing.' "

What Pryor, a moderate Democrat, and Goldwater, the conservative Republican chairman of the Senate Armed Services Committee, did about it was to join forces with other politicians to restructure the military buildup.

Today, Goldwater will be managing the defense spending bill, which calls for limiting the Pentagon's budget growth to only enough to cover inflation, instead of the6 percent after-inflation increase that Reagan sought. The Senate already has voted to cut Reagan's MX program from 200 missiles to 50 and passed several amendments designed to increase efficiency.

"It has begun to stick up here," Pryor said. "Cap Weinberger never did meet a weapon he didn't like. He has suffered so many body blows -- the coffee pot, the toilet seat, allen wrenches, the claw hammer, all this stuff -- that his credibility has suffered severely.

"He should have called somebody in long ago to deal with all these problems and said, 'You've got it.' Then call everybody in and read them the riot act and say, 'This is the guy you're going to be dealing with.' He didn't do that. He has no interest in change or reform. He opposed us on setting up an independent inspector general and an independent operational testing office. His credibility is at its lowest point."

The stories of Pentagon waste have provided ammunition for the budget cutters, but the politicians interviewed said this would not have been enough to turn the tide. There have been other influences:

The menacing cloud of growing federal deficits;

The realization that so much money has been appropriated but not spent that the Pentagon budget would grow by at least 3 percent for the rest of the decade even if it received only enough additional money to offset inflation;

The "fairness issue" -- the conviction that the Pentagon must take a larger share of the cuts in government programs.

Weinberger has tried several tactics to dampen the firestorm but seems to have been unable to extinguish the criticism. He tried exposing fraud by contractors himself, but public skepticism over Pentagon waste continues.

Last week, when it was revealed that the government was paying Grumman Aerospace Corp. $659 for ashtrays, Weinberger approved the transfer of three top Navy officers. Some Navy officials charged that he was looking for scapegoats.

Two of the officers protested publicly that they had assumed their posts after the purchases and decried their treatment. Navy officials have said the officers could be reinstated after an investigation.

"He's been a strong and successful advocate," Pryor said. "Weinberger can sell, but he doesn't inspire. So he's not a Churchill, who could not only inspire in his country but the whole world. He's a salesman and, I imagine, a pretty good poker player.

"Congress has awakened from deep slumber and entered the world of realism," Pryor said. "We realize there has got to be some changes made. Politicians are very sensitive animals. They sense what people are trying to tell them in the supermarket, at the gas station.

"If I were White House chief of staff, I'd keep Weinberger around until this appropriations cycle was over. To move him out now, we'd have to start all over. I'd let it be known that by Thanksgiving we'll have to make a change . . . . "

Sen. John H. Chafee (R-R.I), a former secretary of the Navy, praised the defense secretary for raising the money to buy ships, Army tanks and to raise military pay but faulted him for giving "incredible leeway to the individual service secretaries to proceed with any kind of development they want, without realizing the cost. He clearly hasn't recognized the changing winds and said, 'Let's move together.' He's taken a very hard line."

Other criticisms of Weinberger by senators on committees that oversee the Pentagon were that he has developed no strategy to go with the money he has raised, has overstated the Soviet threat, has failed to build the infrastructure needed to spend defense money wisely and has padded the Pentagon's budget.

Weinberger has his supporters. Goldwater said Reagan's rearmament program is in deep trouble but not because of Weinberger.

"Cap isn't the problem," Goldwater said. "It's the manufacturers; the problem is wrapped up with these manufacturers. It's just not a good season for defense" because of excesses.

Goldwater, Sen. Strom Thurmond (R-S.C.) and Sen. Steve Symms (R-Idaho) said Weinberger should receive high marks for raising record amounts of badly needed money for the services and alerting the nation to the dimensions of the Soviet threat.

"He's for a strong defense," Thurmond said, "so the same people who are against him would be against him anyway."

"It's not as much a Weinberger problem as intellectual flabbiness on Capitol Hill," Symms said.

Pointing to the Senate chamber, Symms added: "Those people in there act like the United States is the enemy, instead of the Soviet Union. People are looking for a scapegoat rather than confront the truth. I give Weinberger a 3.5 grade out of a possible 4.0."

Weinberger said that "the longer one's tenure, the longer the list of people who didn't get what they wanted."

"We're more or less a victim of our success over the last four years in that some people say now it's time to call a halt," he said. "But the threat hasn't diminished. It has increased."

Weinberger said that the nation needs 100, not 50, MX missiles in Minuteman silos and indicated he will renew the fight next year. He said he draws comfort from knowing that both the Senate and House budget resolutions call for returning in fiscal 1987 to 3 percent growth after inflation.

Asked if felt he that should leave his post, Weinberger replied: "We've accomplished a great deal, but there is still a great deal to do. The important thing is to stop before you get tired."

Is he tired?

"No," said the tired-looking Weinberger, with a laugh.