"If an evil empire does exist, let it exist. I'm sure remaking the Soviet Union is not a goal of the United States."

That blunt remark -- playing off President Reagan's bluntest description of the Soviet Union as "an evil empire" -- was made by Soviet leader Mikhail Gorbachev early in his Kremlin meeting with House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) and three other members of Congress last week.

According to one of them, Rep. Silvio O. Conte (R-Mass.), Gorbachev's confident, outspoken style convinced the Americans that Gorbachev was a new-style, even "Westernized" leader.

Conte's detailed notes of that nearly four-hour session provide a vivid portrait of the new Soviet leader.

On new weapons, for example, Gorbachev flatly said the United States "must give up this program" for developing a missile defense in space.

He noted a conflict between the Reagan administration's statements that it "can't give up research" on space defense while simultaneously saying that all issues are "on the table" at the Geneva negotiations.

"You are not talking to Tanzania or Uganda," Gorbachev said pointedly.

He said if the United States continues to develop a space-based defense, his country would push ahead on building additional offensive missiles. These "would cost 20 times less" than the U.S. defensive system, and would push the United States further into debt. Noting American economic difficulties, Gorbachev said: "Maybe we should wait. Maybe you'll want to talk negotiate when things get worse" for the American economy.

On a few occasions Gorbachev used emotion to emphasize a point, leading one of the lawmakers to describe him yesterday as "quite an actor." Gorbachev slammed his fist on the table and pointed to an interpreter as he gave the tough part of his answer -- the United States was "responsible" -- to a question about the recent shooting of an American military officer in East Germany.

As he argued for a moratorium on nuclear weapons construction, he asked almost plaintively, "What do we do with all the weapons we have? Let's stop."

"I get information from Geneva every day," Gorbachev said, speaking of the continuing arms-control negotiations, "and I'm not inspired . . . . What do we see in Geneva today? Marking time. Geneva should not turn into a debate. Otherwise the teams in Geneva will be eating their way through piles of gold rubles, drinking coffee, sipping tea, while mountains of arms continue to be built."

When the Americans raised the subject of human rights, Gorbachev showed the traditional Soviet anger on the subject, saying his country's internal policies were being attacked. "You have your laws," he said, "we have our own laws." Then he added, "I was a lawyer. I studied American law." (Gorbachev graduated from the law faculty of Moscow State University.)

There were a few sparks of what Americans would call political "black" humor. When O'Neill remarked that the relatively little-known Gorbachev had come from out of nowhere to assume power, the Soviet leader responded: "There are lots of places to hide in the Soviet Union."

The Americans present, including the U.S. ambassador to Moscow, Arthur A. Hartman, who was the official note-taker, were impressed by Gorbachev's extensive preparation for the meeting. (Hartman reported on the conversation in a cable to the State Department.)

The session, which had originally been scheduled for 11 a.m., was postponed by the Soviets to 3 p.m., "apparently so he could be better prepared on our positions," one delegation member said.

Gorbachev brought with him sheets of paper on which were paragraphs reporting what members of the congressional delegations had said during earlier meetings on various issues. Some were underlined in green, others in red.

The meaning of the two colors was not clear, but it appeared that he used the Americans' remarks as the outline for his own presentation.

Gorbachev clearly wanted to continue the meeting when O'Neill broke the session off, according to one participant. The delegation had been invited to the ballet that night by the mayor of Leningrad.

On a related matter, U.S. officials revealed yesterday that the Soviet ambassador to the United States, Anatoliy Dobrynin, was called back to Moscow Tuesday. The U.S. officials speculated that he had been summoned to a meeting of the Communist Party Central Committee, of which Dobrynin is a member.

The O'Neill delegation had been told by U.S. officials that a Central Committee meeting this month was expected to fill some open seats in the ruling Politburo. It now contains 10 members, but in the recent past has had as many as 15.

Members of the O'Neill delegation had been told before their trip that adding new Politburo members at the approaching meeting would provide Gorbachev the opportunity to consolidate his leadership position.