The Senate subcommittee investigating Billy Carter met in closed session yesterday to pore over top-secret intelligence reports concerning Libya's efforts to gain influence in the United States.

Chairman Birch Bayh (D-Ind.) told reporters after the 4 1/2-hour meeting that some of the information resulting from U.S. surveillance dealt directly with President Carter's brother, but he refused to say how extensively.

Subcommittee investigators, Bayh indicated, will now attempt to collate the Libyan strategy with various telephone calls and other incidents that have come to light during the inquiry into Billy Carter's dealings with the revolutionary Libyan regime.

Sen. Strom Thurmond (R-S.C.), the subcommittee's vice chairman, said he had not been aware until yesterday's briefing of some of the details gleaned by U.S. intelligence agencies.

Bayh said he had already been briefed on the reports as chairman of the Senate Intelligence Committee, which collected the documentation. He confirmed that "some of that intelligence information is very relevant to our inquiry."

Speaking guardedly after the meeting, Bayh said that much of what was discussed has already been made public, but he was constrained by fears of disclosing the manner in which it was collected. He said some of the "sources and methods" were very sensitive and could be easily jeopardized.

FBI officials testified during the executive session on "the strategy of the Libyans as far as terrorism is concerned," both in this country and elsewhere, Bayh said. He said too much talk about U.S. surveillance and counterintelligence efforts could lead "to an inability to protect ourselves" against such terrorism.

U.S. intelligence reports apparently provided the Justice Department with the first hint of Libya's payments to Billy Carter, which totaled $220,000 by the time he was forced to register as a foreign agent for the Arab government in July.

White House national security adviser Zbigniew Brzezinski also learned from U.S. intelligence reports last spring of Billy Carter's efforts to serve as a Libyan broker with the Charter Oil Co. of Jacksonville, Fla.

In response to questions from reporters, Thurmond said he believed the evidence would show that Billy Carter "was interested" in the C130 military air transports that Libya has been trying to obtain, but Bayh emphasized that that interest on Billy Carter's part was not reflected in the intelligence reports discussed yesterday.

Carter is to appear today before a federal grand jury in Manhattan to tell what -- if anything -- he knows about a friend's State Department briefing on the government's decision to block the sale of the C130s to Libya.

For a year the grand jury has been investigating the tangled affairs of long-time fugitive financier Robert L. Vesco.

The briefing, which was arranged by a White House aide, was given to Henry R. (Randy) Coleman, Billy Carter's friend and business associate, in October 1979, a month before Coleman accompanied Carter to Libya.

Morris Draper, deputy assistant secretary of state, was reported to have told Coleman that Libya was considered a radical nation and probably would misuse the planes.

The grand jury inquiry covers allegations that the Libyans were trying to involve Billy Carter in a scheme masterminded by Vesco to obtain the release of eight transport planes, now parked at a Georgia airstrip.

The material presented at yesterday's closed subcommittee session here included reports from the CIA, the FBI, the National Security Agency, the State Department, and the National Security Council.

Bayh said public hearings would resume next week with testimony from testimony from Justice Department officials.