The Senate Intelligence Committee completed a daylong session with CIA Director William J. Casey yesterday by voting unanimously that it had no basis for concluding that he is "unfit to serve."

Sen. Barry Goldwater (R-Ariz.). the committee chairman, announced the decision in a terse appearance before reporters. He said the committee would still "chase down some of the loose ends" before completing its investigation, but then he abruptly sidestepped a question and refused to answer any more.

"I'm going home," Goldwater announced, turning on his heel and leaving the committee's vice chairman, Sen. Daniel Patrick Moynihan (D-N.Y.), alone at the microphones in a noisy niche of the Capitol crypt.

Goldwater said last week that Casey should resign because he showed bad judgment in insisting on the appointment of a Reagan campaign colleague, businessman Mas Hugel, as chief of covert operations. Goldwater spoke after Hugel abruptly resigned following disclosure of some of his past business dealings.

But most of the committee's 14 other members held back, indicating that they did not regard the Hugel episode as sufficient cause for Casey's ouster and saying that they wanted to wait for an investigation of allegations concerning Casey's long business career.

A preliminary report was submitted to the committee at an executive session Tuesday, but it contained no surprises and reportedly dealt largely with old controversies that had been aired at previous Senate confirmation hearings for other government posts Casey has held.

Increasingly confident. Casey marched into the Capitol yesterday morning for the closed-door session, telling reporters that "it's going to be a cakewalk."

"I've been through this before," he declared.

The session apparently was not quite the "cakewalk" Casey envisioned. It took five full hours and reportedly included prolonged questioning from Sen. Joseph R. Biden Jr. (D-Del.), which one colleague described as a virtual "filibuster."

At the end, Casey backed out of an expected press conference, left the building and slouched into his limousine with a rather tired expression. on his face.

The brief inquiry posed a sharp contrast with last summer's investigation of Billy Carter, with a Senate Judiciary subcommittee carried on for weeks before deciding that it was prepared to question the then president's brother in a productive fashion.

The Casey inquiry seemed to be at a virtual end only two days after the special counsel hired to conduct it; Fred D. Thompson, had started work. t

Asked whether Goldwater had acted too hastily in calling for Casey's resignation, Moynihan avoided a direct answer.

"The chairman of this committee," the New York Democrat intoned, "is a distinguished U.S. senator, a man trusted and beloved in this body, and when he speaks, we respond as we have been responding."

The committee approved its findings in a brief executive session after Casey left. The senators said:

"For the past several days, many allegations have been made about Mr. Casey. Based upon the staff review to date and Mr. Casey's lengthly testimony today, it is the unanimous judgment of the committee that no basis has been found for concluding that Mr. Casey is unfit to serve as DCI," director of central intelligence.

"The staff will follow up on points that need clarification and there will be in timely fashion a final report dealing with the issues concerning Mr. Casey's past activities and also the appointment of Mr. Hugel."

Hugel resigned as the CIA's deputy director for operations July 14 when The Washington Post published an account of his allegedly improper financial dealings in the early 19702. Three days later, the committee formally started a preliminary investigation of allegedly questionable business practices on Casey's part.

Casey denied any wrongdoing. In a July 26 letter to Goldwater and other committee members he contended that all of the questions put to him stemmed from "my activities as a venture capitalist before I entered government service as chairman of the Securities and Exchange Commission in 1971.

Moynihan declined to describe the committee's finding yesterday as "a clean bill of health" for Casey and emphasized that "we have said no more and no less than is in our statement." He also said that the committee's pursuit of the various and unspecified "loose ends" could reverse yesterday's judgment.

"There is a chance of that, yes," Moynihan said. "To say otherwise would be to say we had finished our work."