Defense Secretary Caspar W. Weinberger may have won a battle but lost the war by unexpectedly notifying the Senate Armed Services Committee Tuesday night that he had just found $4 billion in the Pentagon's accounts that could be applied to the 1986 defense budget.

The defense secretary "should have sent it up four months ago," Sen. Barry Goldwater (R-Ariz.), chairman of the Senate Armed Services Committee, said angrily yesterday in discussing Weinberger's belated $4 billion discovery. Asked if Weinberger had lost credibility with the committee and the rest of Congress, Goldwater snapped, "Sure he did."

Other members of the panel, which sets ceilings on how much money the Pentagon can spend to arm and sustain the military forces, conceded that their committee also lost credibility by not finding the hidden $4 billion. Several began publicly displaying their anger at Weinberger yesterday.

When the secretary revealed the existence of this spare cash on Tuesday, the committee was about to begin cutting the 1986 budget. Using the extra $4 billion, the panel rescued some programs it otherwise would have had to slash. The committee was working under instructions from the full Senate to hold the fiscal 1986 authorization bill to the 1985 level, plus enough to cover inflation.

Members of the Armed Services Committee were not happy that Weinberger had advertised to the world that they had missed $4 billion in their supposedly painstaking review of the Pentagon's needs.

Three Republicans -- Sens. William S. Cohen (Maine), Dan Quayle (Ind.) and Pete Wilson (Calif.) -- complained in a letter to President Reagan yesterday that "the manner in which this information has been handed out cannot help but undermine public confidence in the management of our defense resources -- confidence that we hasten to add has been steadily eroded over the last six months by reports of contractor abuses. We who have supported your efforts to see our defense capabilities rebuilt have also seen our credibility suffer as we seek to gain adequate levels of defense funding."

The three urged Reagan "to take positive steps that may be required to to reclaim the public confidence in our defense establishment."

Sen. Edward M. Kennedy (D-Mass.) said Weinberger's crediblity "had sunk to an all-time low" while Sen. Carl Levin (D-Mich.) termed the Pentagon's disclosure of the extra $4 billion "a staggering blow to its credibility."

Sen. Alan J. Dixon (D-Ill.) took the Senate floor and yelled so loud that he could be heard outside the chamber. "This should not be an adversary system," he shouted. "Congress as the people's representative has a right to accurate information. It's not a game."

He said the "flim-flam" with the $4 billion assures that the Pentagon will do just as well under the zero growth-plus-inflation budget as one providing 3 percent real growth. "All our work" on the committee to hammer out a 3 percent budget "was a meaningless exercise. Whatever money the Pentagon wants is apparently squirreled away over there somewhere. And they'll spend it the way they want, no matter what the elected representatives of the people say."

The Armed Services Committee's pride had already been wounded when the full Senate rejected its original conclusion that the Pentagon needed a 3 percent increase above inflation for 1986. The Senate last week voted instead to hold the Pentagon to a "zero growth-plus-inflation" budget. In years past the recommendations of the Armed Services Committee would have carried more weight. But now the Senate, like the House, pays more attention to the Budget Committee, which sets targets for funding for the entire federal government.

Veteran congressional staff members said the disclosure about $4 billion in fat, while far from new for the Pentagon in the annual budget battles, came at a time a growing number of lawmakers were looking for ways to reduce defense spending without appearing to cut into military muscle.

Also, new analyses by the Senate Budget Committee show that the Pentagon has piled up a huge amount of unspent funds that will guarantee many years of increasing defense spending, even if its budget could be frozen at the fiscal 1985 level, plus inflation, from now until 1990. According to the Budget Committee's calculations, even with a freeze spending would continue to grow -- 8.4 percent from fiscal 1985 to 1986, 5.7 percent from 1986 to 1987, 4.5 percent from 1987 to 1988, 4.1 percent from 1988 to 1989, 3.5 percent from 1989 to 1990.

In other words, Congress' generosity over the last four years has already assured sizable growth in defense expenditures for the rest of this decade. As members of Congress have come to appreciate this fact, they have become more willing to cut Weinberger's proposed budgets