Caspar W. Weinberger, whose uncompromising zeal for building up the Pentagon has raised $1 trillion for President Reagan's military buildup, now finds himself under attack from both the left and right, including critics as disparate and unlikely as the Senate Armed Services Committee and the official presidential party in Geneva.

The last five days have been perhaps the most difficult in Weinberger's 58 months at the Pentagon, a longevity record for a Republican secretary of defense. The hostility suggests Weinberger may be in for a long winter, certainly in Congress and perhaps from the White House staff, despite Reagan's continued support for him.

Weinberger's fractious five days began Thursday with a contentious appearance before the usually friendly Senate Armed Services Committee. The tumult continued through last night when his private letter advising the president to avoid concessions to the Soviets in Geneva remained the most controversial document at the summit, largely because a senior administration official portrayed the leaking of the document as "sabotage."

Before the armed services panel, Weinberger said he could support half of the committee's suggested Pentagon reforms. That atmosphere of tentative compromise quickly dissipated as Weinberger turned combative and obfuscatory by turns in sharp exchanges with senators spanning the political spectrum, from conservative Barry Goldwater (R-Ariz.), the chairman, to liberal Edward M. Kennedy (D-Mass.), with a quarrel sandwiched in between with moderate Democrat Sam Nunn (D-Ga.).

In an interview, Goldwater later said of Weinberger: "Well, he hasn't answered a single question that I can recall that Sen. Nunn and I raised on the floor of the Senate . . . . When a man takes more than five minutes to answer a simple question, that's defensive."

Weinberger faces an uphill battle with Congress to keep his future budgets from being slashed to the point that Reagan's arms buildup program will not only level off -- a virtual certainty -- but decline below its present spending level -- a high probability. The secretary also last week contradicted Reagan's endorsement of the Senate version of the antideficit measure known as Gramm-Rudman-Hollings.

As early as Wednesday the House is expected to ignore Weinberger's warnings against changing structure of the Joint Chiefs of Staff by approving legislation giving the chiefs' chairman more power. The Senate is considering more drastic changes.

Another rebuff is expected from the commission Reagan appointed to recommend ways to improve Pentagon management and which is expected to propose fundamental changes early next year. Weinberger opposed creation of this commission, which was appointed amid a spate of well-publicized stories on Pentagon procurement fiascos.

The commission chairman, David Packard, has said of the buildup that "we should have gotten more for our money." Without naming Weinberger and referring to his experience as deputy defense secretary from 1969 to 1971, Packard added, "It's very hard for the guy who's over there in the job to admit he's all screwed up."

Recently, Weinberger has lost most of the big battles in the administration on arms-control issues and he was not invited to accompany Reagan to Geneva. Against Weinberger's advice, Reagan has continued to abide by the limits of the SALT II agreement and has, for now, adopted a narrower definition of restrictions imposed by the 1972 Antiballistic Missile Treaty. Furthermore, Reagan's strategic program is in disarray, raising additional challenges for Weinberger during the rest of his Pentagon tenure. The MX missile, intended as a centerpiece in Reagan's nuclear modernization, has been severely limited by a skeptical Congress. The president recently undercut plans for the Midgetman missile by proposing to Moscow that all mobile missiles be forbidden.

Through all of his travails, Weinberger has hung tough. His contempt for his congressional critics occasionally slips through his usually civil demeanor; last week, for example, he excused himself for being late for a news luncheon on the grounds that he had to take a telephone call from a congressman demanding more defense work in his home district while also demanding Pentagon budget cuts.

Weinberger has bought a house in McLean and shows no sign of returning to California. He has said he will stay at his post as long as Reagan wants him, which could be a long time judging from the president's comment Sunday when asked in Geneva whether he would fire Weinberger. "Hell, no," he replied.