The House Assassinations Committee was cautioned by oen of its members yesterday to settle down and avoid any semblance of "McCarthyism" in its investigation of the murders of President Kennedy and the Rev. Martin Luther King Jr.

Rep. Charles Thone (R-Neb.) told his colleagues at an unannounced committee meeting that he felt "we've got to lean over backwards to be responsible" now that the House has voted to keep the inquiry going through 1978.

In the floor debate preceding that vote last week, some critics of the inquiry charged that some of the committee's tactics smacked of the McCarthy era, especially in the public hearing it held last month with Florida mobster Santo Trafficante Jr. as the only witness. Trafficante refused to testify, invoking the Fifth Amendment and other constitutional guarantees.

Citing recent newspaper editiorials and television commentary about the inquiry, Thone said he felt the committee had to be "terribly, terribly careful" that it does not lapse into McCarthyism.

McCarthyism refers to a period in the early 1950s when the late Sen. Joseph McCarthy presided over a series of hearings based largely on unsubstantiated charges that the federal government had been infiltrated by Communists.

"We've got to settle down," Thone said. "We must be careful we don't have anything that could possibly be called grandstanding."

The Committee had planned to meet secretly in what it calls an "informal" session, but was reportedly advised that it had to convene publicly at first in order to take formal action accepting two resignations - those of special counsel Alvin B. Lewis Jr. and security officer Frank Kelly.

Lewis, a Pennsylvania lawyer, told reporters later that he was leaving primarily because of the abrupt resignation last week of chief counsel Richard A. Sprague. Sprague quit in the face of strong signs that the house would have refused to keep the inquiry going unless he stepped down.

"I have the greatest respect for Sprague," Lewis said. "I think he's the best man in the country for the job. That's why I came and that's primarily why I'm leaving."

Kelly served notice weeks ago that he would be leaving to return to an earlier jon with the Senate Intelligence Committee.

Over Thone's protests, the committee then voted 5 to 1 to move into secret session to discuss the expected course of the investigation over the next 30 days and the mechanics of choosing a new chief counsel.

Thone said he saw no reason for a closed-door discussion of possible methods and deadlines for picking a new counsel.

The committee is seeking a $2.7 million budget for calendar 1977, but will have to wait until April 19 for a hearing on the request before the House Administration Subcommittee on Accounts.