William J. Casey, the embattled director of the Central Intelligence Agency, vowed to fight for his job yesterday, but support for him was fading fast on Capitol Hill and within the administration itself.

Despite the mounting opposition, however, Casey had a powerful patron on his side. Late in the afternoon President Reagan told reporters that "we still have confidence" in the CIA director, who is the target of a Senate Intelligence Committee investigation into past financial practices.

Nonetheless, the administration has carefuly laid the groundwork for a change in tis position if the Senate committee caomes up with a recommendation that Casey should go. White House spokesman David R. Gergen issued a statement yesterday pledging full cooperation with the Senate committee, but added: "Everyone recognizes that the conclusion of that committee may affect the climate on the Hill and the climate elsewhere."

Reagan declined to say whether the administration would necessarily abide by the committee's recommendations but said he would talk to the senators about whatever they recommend.

Other White House officials said they were concerned that Republican opposition to Casey could damage his effectiveness and make it difficult for him to continue as head of the CIA.

On Capitol Hill, two more leading Republican senators, Ted Stevens of Alaska and William V. Roth of Delaware, declared that Casey should resign. This development followed Sen. Barry Goldwater's announcement at a Thursday night press conference that he felt Casey should quit because of his appointment of Max Hugel as his clandestine service chief. Hugel was forced to resign because of a financial scandal.

Goldwater (R-Ariz.) is chairman of the Senate Intelligence Committee. His announcement came as a shock to administration officials, who thought Goldwater was going to refute a television report that he had privately asked Casey to quit.

Yesterday, Senate Majority Whip Stevens and Roth, a member of the Senate Intelligence Committee, aboth told reporters in outspoken terms that Casey should resign.

"He should go -- now," Roth said emphatically.

A member of the Senate defense appropriations subcommittee, Stevens said that panel, too, was "worried about the future of the agency if the director becomes the focal point of controversy at this time.

"It's my judgment that Barry doesn't make these recommendations lightly," Stevens added. "He has the interests of the agency at heart."

Roth said at a press conference that he felt strongly that Casey's effectiveness had been compromised already.

"The director of the CIA must be above suspicion, and to borrow a phrase from President Eisenhower, 'cleaner than a hound's tooth,'" Roth said. He said the Intelligence Committee's investigation into allegations involving Casey's pre-CIA business dealings was continuing, but he felt that Casey's credibility with the committee had been so damaged already "that I believe it is impossible for Mr. Casey to effectively discharge his duties."

The opposition of Stevens, who said that Casey should leave "for the good of the agency," shook White House officials. Stevens has a reputation at the White House as a highly dependable and thoughtful senator who rarely strays off the party reservation, and his opposition was viewed as a signal that Casey is in deep trouble.

The growing criticism of Casey in Congress was laced with expressions of concern for the agency and repeated reminders of the delicate state of international affairs. What seemed to be bothering the senators was not any hard evidence of wrong-doing on Casey's part but lack of confidence in his judgement as head of the entire U.S. intelligence community.

"Everyone makes fun of him up here," said one well-attuned congressional source. "There's a feeling that he's got all those harebrained schemes that he's too willing to go along with. From time to time, CIA people will say, 'Wait till you hear what we had to talk him out of this time.'"

Opinion is more divided within the administration, but the same concerns have been raised there, too. One source was critical of Casey's "lack of contributions" at National Security Council meetings. Another thought that Casey was "getting a bum rap" but was puzzled at the depth of Senate opposition, which is taken seriously at the White House.

There is no desire at the White House for conflict with a GOP-controlled Sneate that has supported Reagan's major legislative proposals.

"Before that happens, I'm sure the president will sit down with the senators and talk it over," an administration official said.

Reagan did not talk to either Casey or the Senate critics yesterday. That was left to White House chief of staff James A. Baker III, who is overseeing the administration strategy on Casey. White House counsel Fred Fielding is reading documents as they are submitted to the Senate committee, and Gergen said that, so far, "nothing has come to light which has changed our original evaluation."

Casey used a private office in the Executive Office Building yesterday as he set about preparing his defense. Gergen said that to the best of his knowledge he did not meet with White House officials.

The CIA director spent much of his day making the rounds of Senate offices, seemingly determined to ride out the storm. But to reporters who tracked his moves he said little beyond advising them to "read my statement."

In it, he dismissed as "absolutely false" a report, aired by Goldwater Thursday night, that Casey might have made $750,000 out of a now-defunct agribusiness called Multiponics instead of having lost money in the venture.

Casey said he believed materials that he plans to provide to the Intelligence Committee Monday "will lay this entire controversy to rest."

"We are cooperating with the committee and will continue to do so should additional questions be raised," the CIA director said.

He concluded the statement by saying he was looking forward "to a continued close and productive relationship with Sen. Goldwater" and other members of the House and Senate Intelligence committees "as we work together in the months ahead."

One senator who asked not to be named said Casey sounded much less determined in a conversation yesterday. Asked if Casey was bent on staying in the job, this senator said, "I got the opposite feeling."

Senate Majority Leader Howard H. Baker Jr. (R-Tenn.) said yesterday morning that he had talked with Goldwater and "I want to back up Goldwater in whatever he decides to do . . . I am confident he will handle the matter appropriately."

Baker, who met with Casey later pointedly telling reporters at one of his periodic press luncheons that "Mr. Casey would be wise to accept Mr. Goldwater's advice."

The Alaska Republican said he had talked with members of both parties on the Intelligence Committee and got "the firm impression that they have good reason" to want Casey out, "because of matters of judgment more than anything else."

Roth called CIA deputy director Bobby R. Inman "a very well-qualified man" who could easily move up a notch if Casey should leave.

In fact, Inman was the first choice of Goldwater and most members of the House and Senate Intelligence committees before Reagan picked Casey last December.

United Press International reported last night that Goldwater has offered Fred Thompson, Republican counsel to the Senate Watergate Committee in 1973, the job of counsel to the Intelligence Committee for its Casey investigation. Thompson, described as a Nashville lawyer with close ties to Majority Leader Baker, was quoted as saying, "I hope to have a decision made by Monday."

Thompson also was counsel to the Senate Foreign Relations Committee this year for tis background investigation of Alexander M. Haig Jr. to be secretary of state. Questions were raised about the nominee's role in the Watergate coverup as then-president Nixon's chief of staff.