The Reagan administration yesterday cooled its rhetoric about the latest confrontation in Lebanon between Israeli forces and U.S. Marines as influential lawmakers expressed deep concern and raised the possibility of an on-the-spot congressional investigation in Beirut.

The State Department, through spokesman Alan Romberg, said it stood on its Wednesday statement that three Israeli tanks had attempted "to cross into territory" under control of Marines in a "recurrence of challenges" which the U.S. government views "very seriously."

But he did not repeat the charges or elaborate on them.

"There clearly is a need for clarification of the situation to avoid future incidents, and we are working on it," Romberg said.

He would say little else as he was peppered with questions about Israel's denial that it invaded Marine territory, about Pentagon rules forbidding Marines to talk to Israeli counterparts, and about alleged new arrangements in Beirut on whose forces should control what territory.

"We thought the situation was clear," Romberg said of the operational areas for U.S. and Israeli forces. "Obviously it wasn't."

That response contrasted sharply to his statement of 24 hours earlier that "such incidents endanger the safety of the troops involved and hamper the peacekeeping efforts of the multinational force."

Romberg did add a bit of official commentary on this latest and most serious incident, in which a Marine captain jumped on the Israeli commander's tank with a loaded .45 pistol to stop the advance. "My understanding is that some effort was made" to use the hotline set up for such confrontrations "but it was an unsuccessful effort," he said."

Senate Majority Leader Howard H. Baker Jr. (R-Tenn.) said yesterday that he had conducted his own inquiry by talking by telephone to Israeli Ambassador Moshe Arens, who was in California, and acting Secretary of State Ken Dam. Baker also met in his office with Gen. Paul X. Kelley, assistant Marine Corps commandant.

"I was concerned about sending troops to Lebanon initially, and said so publicly, and I continue to be so," Baker said during lunch with several reporters.

In the House, Rep. Dan Daniel (D-Va.), chairman of the Armed Services subcommittee on readiness, who had reacted angrily on Wednesday to the Israeli tank incident, said yesterday that he is considering going to Beirut with some fellow members to investigate U.S.-Israeli military relationships in Lebanon.

"The idea would be to reassess the situation and see what we could do to help stabilize it," Daniel said in a telephone interview. He added that on his last visit to Beirut on Oct. 4 he found the relationships between Marine and Israeli forces "extremely tense."

"When the Israelis left the Beirut airport, they only moved back a hundred or so yards," Daniel said in describing what he saw then. "It was not unusual for the Marines to find these Israeli guns trained on them. I saw them.

"Israel has got to understand that the members of Congress are not going to appropriate funds if these type of incidents continue," said Daniel. "I've always been a very strong supporter of Israel, but this is just intolerable. It wouldn't surprise me if these incidents do continue, because the attitude over there is just not good."

Daniel, like Baker, plans to meet with Assistant Marine Commandant Kelley. The focus of their conversation is expected to be how Marine-Israeli relationships on the ground can be improved to reduce the chance of future confrontations.

Defense Secretary Caspar W. Weinberger told the House Armed Services Committee on Wednesday that he fears that the recent incidents, which he said were provoked by Israeli military actions, "will escalate to serious loss of life."

Howard W. Squadron, president of the American Jewish Congress, had this comment yesterday on Weinberger's congressional testimony:

"The anti-Israeli theatrics of our secretary of defense over a minor incident in Lebanon is dismaying. . . .

"Rushing to a Senate [sic] committee to escalate an insignificant happening on the ground into an international incident of major importance makes us question his sense of proportion as well as his motives.

"There is every reason to believe that the secretary's account of the affair was inaccurate and distorted....

"What the secretary has attempted is to deliberately exploit the confusion that necessarily exists in a military zone where respective responsibilities of the various parties have not yet been settled or clarified. His behavior is inexcusable . . ."