The Senate opened hearings yesterday on the Biily Carter-Libyan connection as both Republican and Democratic senators sharply questioned the propriety of using the president's brother as an intermediary in the Iranian hostage crisis.

Undersecretary of State David Newsom defended the White House's use of Billy Carter as "an appropriate presidential initiative in a time of problems." He told the first session of the Senate investigation subcommittee: "One cannot exclude in the Arab world a certain respect for family connections."

Sen. Patrick Leahy (D-Vt.), however, questioned whether Billy Carter's "training as a gas station operator" qualified him for sensitive dealings with the Libyan government. The president's brother was asked last winter by National Security Affairs adviser Zbigniew Brzezinski, on the presdient's suggestion, to arrange a meeting with Libyan diplomat Ali Houderi and seek Libyan help in resolving the hostage situation.

Sen. Richard Lugar (R-Ind.) asked whether Billy Carter was a "credible intermediary" in the hostage crisis. Sen. Bob Dole (R-Kn.) inquired sarcastically whether, after Billy Carter's meeting, "something happened that was useful besides the burning of the embassy."

The U.S. Embassy in Tripoli was burned by a mob on Dec. 2, 1979, a few days after Billy Carter's Nov. 27 meeting with Brzeninski and Houderi. f

Newsom, however, said of the meeting, "I can, as a professional diplomat, understand the feeling of someone in a position of responsibility that every avenue should be explored to see if the lines of communication could be opened."

The four-hour hearing was designed, as one senator put it, to be "a history lesson" on the rocky relationship between the United States and Libya, a strategically located North African country which supplies America witht 10.8 percent of its imported oil.

The United States has frequently condemned the Libyan government for encouraging terrorism abroad. And, Newsom said, U.S. diplomats withdrawn from Tripoli after the embassy attack will not return until Libya can assure their safety.

Newsom told the subcommittee, however, that despite cool relations, he met with Libya's ambassador to the United Nations four days after the hostages were seized in Tehran to urge Libya to "take a stand against the seizure of the hostages . . . [as] part of our global effort to mobilize international opinion and pressure."

That meeting, on Nov. 8, 1979, occurred well before Brzezinski, on Nov. 20, asked Billy Carter to set up the meeting with Houderi. While Newsom's earlier plea to Libya raises the issue of whether Billy's role was necessary, Newson said yesterday, "We cannot exclude the use of persons who have certain kinds of contacts."

While the hearing, in a high-ceilinged, wood paneled room in the Dirksen Office Building lacked the drama of the 1974 Watergate investigation, it had a few of the trappings: 11 television cameras, more than 30 news reporters and, waiting in long lines in the corridor, some 300 would be spectators drawn by the whiff of scandal.

"It's history, I guess," said Kathy Spencer, 24, who moved here this summer to work as a Labor Department attorney. "I want to see it firsthand. sAt first I thought it was Billy's buffoonery. But as it became apparent that the State Department and Carter's advisers were giving Billy help -- that shocked me."

William Downs, 42, a tomato farmer from Corpus Christi, Tex., waited in line dressed in jogging shoes and a yellow T-shirt inscribed, "Willie Nelson's Distance Classic." Here as a tourist, Downs came because "it's a happening." Nonetheless he sees the Billy Carter affair as "a tempest in a teapot." A registered Democrat, he said he plans to vote for independent candidate John B. Anderson.

Another would-be spectator, Lilo Ross, 70, of Pittsburgh, said, "If Billy had any qualifications, it would be different. But he's a typical southern redneck." Besides, she added it was unseemly for the president's brother to make money off a regime led by Col. Muammar Qaddafi, whom she described as an "anti-American, criminal and very dangerous."

Newsom's testimony underscored the problems of dealing with Quaddafi who he said, "has in recent months publicly called upon Palestinian groups to attack Egyptian, Israeli, and American target's in the Middle East." Newsom added that the Libyans, "have provided money, training and in some cases, arms to virtually any group around the world which asserts revolutionary credentials, including the Moro insurgents in the southern Philippines, the Provisional Wing of the Irish republican Army, the Japanese Red Army and certain African organizations."

Newsom also cited a series of assasinations of Libyan citizens in Europe, in an officially sanctioned effort to stifle opposition to Quaddafi, and an intimidation campaign against Libyan citizens and student dissidents living in the United States.

Nonetheless, he said, the United States has "Important reasons for seeking to find a basis for satisfactory relations," not the least of which is $9 billion in annual oil purchases, the presence of 50 American companies in Libya, most of them in oil-related enterprises, and the fact that Libya has threatened to embargo oil sales to the United States because of political differences.

The senators onthe subcommittee, led by Chairman Birch Bayh (D-Ind.) and ranking Republican Strom Thurmond (S.C.), went out of their way yesterday to stress that the committee, despite obvious partisan temptations in an election year, will be "responsible" and "objective."

An importanat indication will be the choice of the subcommittee's chief counsel. While no final decision had been made yesterday, the front-runner seemed to be James Neal, the former Watergate prosecutor, who, Dole said, would bring "credibility" to the effort and "neutralize any patisan overtones."

The committee has already hired prominent former FBI agent John McDermott as the chief investigator.

Thurmond told the hearing, "I want this investigation to be fair, professional and nonpartisan . . . Whatever others may do, the Senate will not be guided by sensationalism or desire for headlines."