All six members of the Senate Select Committee on Ethics have told Senate party leaders that they plan to quit the controversial panel as soon as possible, senior committee members said yesterday.

At least three of the committee's members, however, have agreed to remain until the current committee investigations of Sen. Herman E. Talmadge (D-Ga.) and former senator Edward W. Brooke (R-Mass.) are concluded. The three include Chairman Adlai E. Stevenson (D-Ill.), Vice Chairman Harrison H. Schmitt (R-N. Mex.) and Robert B. Morgan (D-N.C.).

Stevenson and Schmitt, meeting with reporters yesterday, said the ethics panel hopes to wrap up its final report on Brooke next month after an appearance in closed session by the former senator, and begin its investigatory hearing into allegations of financial wrongdoing by Talmadge in March.

The committee met yesterday to begin discussion of a preliminary report on Brooke that was completed this week. Stevenson said Brooke is scheduled to appear in person before the panel next week and a final report will be issued on the former Massachusetts senator after that.

The committee also heard from Talmadge's Washington attorney yesterday in a plea to exclude major evidence from its investigation into alleged financial misconduct by Talmadge.

The Georgia Democrat checked into Bethesda Naval Hospital Monday night, claiming through a spokesman that he was suffering from exhaustion and alcohol abuse. His attorney, James Hamilton, told the committee yesterday during a closed session that he did not expect Talmadge's hospitalization to delay the committee's planned hearings.

Talmadge has sought to have evidence collected by his former key aide Daniel Minchew excluded from the committee's hearing. He has claimed the material -- mostly financial records -- was taken from his office by Minchew without his consent.

Stevenson told reporters that he has consulted with Senate Majority Leader Robert C. Byrd (D-W.Va.) about the future of the committee and told Byrd he and the two other committee Democrats, Morgan and Sen. Abraham A. Ribicoff (D-Conn.), want to leave the panel.

Stevenson said he agreed with Byrd's request that he remain on the committee on a temporary basis. He said the successor to Ribicoff, the only committee member who voted against continuing the probe of Talmadge, will take over the chairmanship of the committee.

Schmitt said all three Republican committee members have indicated their wish to leave the panel. But he said Senate Minority Leader Howard Baker (R-Tenn.) has not decided who will be allowed to resign.

According to knowledgeable Senate sources, there has been little enthusiasm among the remaining senators to serve on what is one of the most unpopular of committee assignments.

Committee rules require the rotation annually of one Democrat and one Republican. The Democrats also agreed when the committee was formed in 1977 that none of the current Democratic members would serve beyond 1973. The Republicans have no such agreement.