President Carter yesterday ordered all members of his staff to "cooperate fully" with a special Senate subcommittee set up to investigate his brother's ties to Libya, and left open the possibility that he would testify before the panel, if necessary.

In a statement in response to creation of the special subcommittee of the Senate Judiciary Committee, White House officials said Carter "does not expect to assert claims of executive privilege with respect to those matters." They said that all members of the White House staff, including national securitiy adviser Zbigniew Brzezinski, will testify under oath before the subcommittee, if called.

The statement was a sweeping pledge of cooperation by the White House, Presidential aides clearly hope this will lead to a speedy resolution of the controversy before it inflicts serious political damage on the president.

Meanwhile, the question of Billy Carter's ties to Libya took another bizarre turn yesterday when White House press secretary Jody Powell disclosed that First Lady Rosalynn Carter may have played a role in recruiting her brother-in-law as a contact with Libyan officials.

Powell said that Mrs. Carter recalls that some time after the American hostages were captured in Iran last Nov. 4 she called Billy Carter, who by then already had a well-publicized connenction with Libya, and asked him whehter he thought he could help in trying to win the release of the hostages.

Powell said Mrs. Carter recalls that her brother-in-law's response was "positive," and that she believes her call to Billy Carter came after Brzezinski had made a similar request to him that led to a Nov. 27 meeting of Brzezinski, Billy Carter and Libyan diplomat Ali Houderi.

Mrs. Carter's role was the latest twist in the Billy Carter controversy, which began last week, when, under pressure from the Justice Department, he registered as an agent of the Libyan government. In the registration, Billy Carter disclosed that he has received $220,000 in payments from Libya.

He said these payments were installments on what was to have been a $500,000 loan from Libya to him. The Justice Department declined to accept his characterization of these payments as a loan.

The Billy Carter matter yesterday preoccupied Powell and other senior presidential assistants, who are clearly concerned about comparisions between their handling of this controversy and the behavior of the Nixon administration during the Watergate scandal.

Pledging to be forthcoming and cooperative with any investigation, Powell said, "Judge us by what we do." He added:

"We believe we will come out all right in the end because we have behaved in a proper manner in this matter and that the best way to make that clear to the American people is to provide them with as much information as possible."

The White House statement on the Senate subcommittee pledged complete cooperation with the investigation in three areas. It said the White House will honor "requests for information about the relationship between Billy Carter and the government of Libya, as well as about any contacts between any member of the White House staff with Billy Carter or with the Department of Justice relating to Billy Carter."

The statement did not rule out a claim of executive privilege if questioning strayed from these three areas, but Powell said that based on public statements of members of the subcommittee there is no reason to think that executive privilege will become an issue.

The question of whether the president is willing to testify before the committee was left open in the White House statement, but Carter did pledge his personal cooperation.

"The president himself will also respond fully to the subcommittee's inquiries relatiang to these matters, in accordance with mutually agreeable procedures consistent with the responsibilities and time constraints of his office," the statement said.

Powell said that since there had been no discussion with the subcommittee of what such "mutually agreeable procedures" might be, he could not say whether they might involve a personal appearance before the panel by Carter.

There is one precedent for a president testifying before a congressional committee: former president Gerald R. Ford did so to explain and defend his blanket pardon of his predecessor, Richard M. Nixon.

But in offering to make Brzezinski available for sworn testimony before the subcommittee, the White House appeared ready to break a precedent. National security advisers to presidents traditionally have not formally testified on Capitol Hill, and presidents, including Carter, have resisted efforts to make their national security advisers subject to Senate confirmation, which would then require them to testify periodically.

Powell spent most of yesterday in meetings with other presidential assistnats on the Billy Carter matter and in briefings with reporters on the controversy. In the course of this, he amended slightly some earlier accounts of contacts involving the White House, the president's brother and Libyan officials.

Powell said that a meeting last Dec. 6 which involved the president, Brzezinski and Houderi did not, as he had first told reporters, come about at Houderi's request. He said the president asked Brzezinski to arrange the meeting so that he could inform the Libyan diplomat personally of his displeasure over the sacking of the American Embassy in Tripoli.

Powell took personal responsibility for the lack of any mention of this meeting in the "white paper" the White House released earlier this week on contacts involving Libyan officials. He said he excluded it because he thought at the time there might have been more than one meeting between Houderi and the president and that he wanted to check that before saying anything. He said yesterday that Dec. 6 was the only Carter-Houderi meeting.

Powell also said that Brzezinski held another, separate meeting with Houderi on Dec. 12. He said Houderi requested this meeting to report his government's response to the president's Dec. 6 protest over the embassy burning.