The Senate Intelligence Committee concluded its prolonged investigation of CIA Director William J. Casey yesterday with a slender report that members said faults him for inattention to detail but finds no cause to call for his ouster.

The report, which is less than 10 pages long, will not be released until today, after Casey has had a chance to review it.

"It's safe to say the whole situation is not very flattering," Sen. Harrison H. Schmitt (R-N.M.) told reporters after an executive session yesterday afternoon.

"But you can't come to the conclusion that he is unfit to serve . . . . I just wish he was more attentive to details in his personal life."

According to several senators, the bottom line of the report is this negatively phrased finding that Casey is not "unfit to serve" as director of the Central Intelligence Agency.

The investigation, which lasted four months, delved primarily into Casey's business dealings and his failure to disclose various aspects of them in the financial reports he made to the Senate when he was nominated CIA director. "There were omissions," Schmitt said. "I'm convinced they were inadvertent, but there were omissions."

Several committee Democrats were less sympathetic. Sen. Walter D. Huddleston (D-Ky.) said, "My personal conclusion is that he has not been that inattentive, that it has been by design, maybe. I raise that as a question."

Huddleston said he also thought the report, which the committee accepted in place of a longer and more detailed version, still contains enough evidence for President Reagan to conclude that it might not be in the best interests of the CIA to keep Casey on.

"The ball is back in the president's court," Huddleston told reporters. "It's his appointee. I would think that an administration that has said much about strengthening intelligence ought to give consideration to whether it ought to continue" with Casey in charge.

Sen. Patrick J. Leahy (D-Vt.) said that "with one or two exceptions" on the committee "certainly people were not pawing over one another to support Mr. Casey."

Leahy said he personally felt that "the questions raised here in the Casey inquiry were a heck of a lot more serious than the questions raised about Reagan's national security adviser Richard V.Allen."

Allen came under Justice Department investigation for his receipt of $1,000 tendered by a Japanese magazine last January following a brief interview at the White House with Mrs. Reagan and allegedly intended as a thank-you gift for her.

Leahy said he was more concerned about allegations that Casey had misled potential investors in a defunct agribusiness firm and his failure to list all the sources of income required under Senate rules for presidential appointees subject to confirmation. Leahy said some committee members no longer had any confidence in Casey's credibility.

Schmitt, however, said that part of the problem lies in what he called "the ethics-in-government environment" that lawmakers have created in recent years. "We have made it unbelievably complex," he said.

"The main thing we found is that he was not attentive to detail in his own business matters," Schmitt said. But he argued that this should not be held against Casey in his capacity as CIA director anymore than the fact that "you don't make your bed in the morning means you're not going to keep a clean desk at the office."

At the CIA, Schmitt maintained, Casey has improved morale and "is demonstrating his fitness every day" in the intelligence field. The New Mexico Republican said there was no question in his mind that Reagan would keep Casey on the job.