The international dispute over Soviet troops in Cuba took a new turn yesterday with intensified deliberations by President Carter and several sets of advisers, and renewed attacks on the U.S. position from Havana and Moscow.

The White House, which took pains to call the diplomatic impasse a problem rather than a crisis, announced that Carter will address the nation at 9 p.m. Monday to explain the controversy and his decisions about it.

Carter attended part of a three-hour afternoon meeting of his senior foreign policy aides. White House officials said a draft of his proposed Monday night statement was being discussed.

Carter also met with the National Security Council for an hour Thursday night and nearly two hours at breakfast yesterday, after the return of Secretary of State Cyrus R. Vance from apparently unsuccessful negotations with Soviet Foreign Minister Andrei A. Gromyko in New York.

Vance canceled a Yale University speech today and a trip to Panama scheduled to begin Sunday in order to participate in policy discussions here.

A group of about 12 former high government officials, modeled on the "wise men" who advised President Lyndon B. Johnson on Vietnam policy in 1967-68, met last night to consider the Soviet brigade issue and discuss recommendations to be made to Carter. Another meeting of the outside advisers is planned for today.

Carter previously announced that he will take unspecified "appropriate action" if negotiations with the Soviets fail to change the "status quo" in Cuba. The U.S. position is that a newly discovered Soviet "combat brigade" of 2,000 to 3,000 men in Cuba is unacceptable, although top officials have also said it does not threaten the United States.

The Soviet Union's position is that its force in Cuba is a "training center" that has not changed in its number or functions in the past 17 years, and does not violate any U.S.-Soviet agreements. An apparent Soviet refusal to shift ground despite the U.S. demand has brought the present impasse.

Among the compensatory U.S. actions under discussion, according to informed U.S. officials, are ways to display U.S. military strength in the Caribbean without calling forth an automatic Soviet response leading to a higher level of tension. Possible actions under discussion, according to the sources, include:

U.S. naval exercises in the Caribbean to show the flag in more dramtic fashion.

A resumption of overflights of Cuba by U.S. spy planes.

Announcement of an "agonizing reappraisal" of U.S. policy in the Caribbean.

A proposed increase in U.S. intelligence budgets to display U.S. determination to upgrade its surveillance of the area.

Highly punitive recommendations on a "tough list" of proposed measures have lost out in the policymaking so far, according to sources.

National security affairs adviser Zbigniew Brezezinski, interviewed by National Public Radio, compared the dispute over the Soviet brigade with the confrontation over the building of the Berlin Wall in 1961, rather than with the Cuban missile crisis of 1962.

Soviet erection of the wall between East and West Berlin "was an unacceptable situation . . . [which] continues to this day," Brzezinski said. "Because it was unacceptable, President Kennedy had to take certain measures to reassure the West Berlin population, to reassure our allies, indeed to indicate to the other side that we took a very grave view of that."

He added that "as a political problem" the dispute over the Soviet brigade in Cuba fits the 1961 analogy more closely.

In response to the building of the Berlin wall, President Kennedy requested a $3.2 billion increase in the military budget, reinforced the U.S. garrison in Berlin, called up 85,000 reservists to active duty and sent then Vice President Lyndon B. Johnson and former U.S. military commander Gen. Lucious Clay to Berlin.

Despite the deepening of the dispute over the brigade, Carter is expected to continue to call for Senate approval of the strategic arms limitation treaty (SALT II) in his Monday address. Abandonment of SALT by the administration or its defeat in the Senate could generate greater shock waves in the Soviet Union and among U.S. allies, and force new changes in U.S. strategic programs.

Senate Majority Leader Robert C. Byrd (D-W.Va.) appealed to the Senate yesterday to permit SALT to be considered on its own merits. "It should not become the hostage or the victim of a pseudo crisis," Byrd said.

He was responding to a proposed amendment by Sen. Nancy Kassebaum (R-Kan.) to prevent the treaty from taking effect until the president certifies that no Soviet combat troops are in Cuba.

A group of 52 members of the House of Representatives headed by Rep. Paul Simon (D-Ill) released a statement urging the Senate to consider approval of SALT II without a link to the Soviet troops issue.

The American Committee on East-West Accord, which backs SALT II, issued a statement calling the dispute over the Soviet brigade in Cuba "a false crisis over a nonevent" and expressing concern that senators are finding it "politically irresistible" to link it to SALT II.

Despite the flurry of meetings and new charges from the communist capitals, the White House continued to avoid new public statements. There was no White House briefing for reporters, an event that is usually held every week day.

The State Department continued to say nothing official about the status of the negotiations.

Members of the "wise men" advisory group anticipate that they will meet with Carter to discuss their findings before his address Monday night. While no format has been specified, the expectation is that the former officials will have an opportunity to express their views on the dispute and U.S. counteractions as well as contribute to a possible consensus that the president may be able to cite publicly.