The United States is asking the Soviet Union to withdraw its combat troops from Cuba as negotiations on the matter enter the crucial decision-making phase, President Carter reportedly told key lawmakers yesterday.

According to participants in the White House briefing for senior members of Congress, Carter added that a neat and complete withdrawal of the Soviet force is unlikely, given the complexity of the situation.

If the problem cannot be solved through negotiation, Carter continued, the United States is prepared to take "offsetting and compensatory" actions of its own. Administration officials said such U.S. contingency options, which have been under intensive discussion at the White House, would not be in the nature of a military threat. The possible countermoves under consideration were not disclosed to the lawmakers.

The president appeared to be upset, according to participants, with Soviet statements that it has only a long-standing military training center in Cuba, rather than a combat brigade as the United States insists. At one point in the briefing, Carter is reported to have charged that Russians "lied" to the United States about the mission of the troop unit.

The White House briefing, which provided the most detailed report to date on the difficult and sensitive negotiations, came several hours before Secretary of State Cyrus R. Vance met Soviet Ambassador Anatoliy F. Dobrynin in their fifth round of talks on the issue at the State Department.

An authoritative Carter administration official told reporters that the discussions up to yesterday were primarily a fact-finding exchange, with the United States asking "a series of very specific questions" about the Soviet force in Cuba, and the Russians providing replies after consideration "at a very high level" in Moscow.

"We now have as much information as it is possible to derive" from the diplomatic exchange and from a thorough analysis of data available over the years to U.S. intelligence, according to the official, who declined to be quoted by name.

The negotiations are now entering a new phase in which Washington and Moscow will begin discussing steps "to relieve -- or to alter -- the situation in a way that would be acceptable to the United States," the official continued.

Carter is reported to have told the legislative briefing that the United States hopes for an answer from Moscow by the end of this week to the position that was presented by Vance to Dobrynin in their meeting yesterday.If this is not forthcoming, Vance will take the issue to Soviet Foreign Minister Andrei A. Gromyko in a meeting expected to take place late next week in New York, where Gromyko is attending the United Nations General Assembly.

With Senate approval of the strategic arms limitation treaty (SALT II) hanging on the successful resolution of the troops-in-Cuba issue, several of the lawmakers urged Carter to move quickly. After requests from the legislators, he agreed to brief them again in about a week.

Although it was widely assumed that the United States would ask for withdrawal of the 2,000 to 3,000 Soviet troops, until yesterday high officials had avoided any description of their objective in the negotiations other than to say with deliberate ambiguity that "the status quo is not acceptable."

Carter's remark about the difficulty of achieving the U.S. goal in neat and complete fashion seemed to hint at possible solutions well short of the maximum. One legislator said he obtained the impression that the Soviets have already ruled out a total withdrawal of what the United States describes as the combat unit.

The Soviets have publicly insisted that their force in Cuba is unchanged in its number or function over the past 17 years. A number of U.S. intelligence analysts now accept as a real possibility the Soviet Union's claim that its force has been in Cuba that long, but U.S. intelligence continues to insist that at least part of the force is organized and equipped as a combat unit.

Both Carter and his national security affairs adviser, Zbegniew Brzezinski, referred during yesterday's briefing to a 1969 fact-finding report to then President Nixon by Nelson A. Rockefeller, who was then governor of New York. After an official trip to Latin America, Rockefeller reported the presence of about 6,000 Soviet troops in Cuba. Some of the lawmakers took this reference to be a Carter administration suggestion that a Soviet combat force in Cuba dated back at least to 1969.

Much of yesterday's meeting was consumed by the comments of the legislators, including suggestions that Carter take a tough line in the negotiations and lawmakers' estimates of the impact on approval of SALT II.

By bringing Congress abreast of the negotiations, Carter was able to obtain firsthand a better sense of the legislative mood and desire as hard bargaining with the Soviets begins. Carter was also clearly bidding for bipartisan backing for his position, and for the forebearance of the lawmakers while Vance pursues the U.S. objective in the secret talks.

It was also useful, from the White House viewpoint, that several lawmakers made strong public statements immediately after the meeting emphasizing the political gravity of the problem of the Soviet troops. Any chance that the Soviets will accommodate Washington is believed to rest on a calculation that such action is required in the interest of SALT II and the continuing working relationship of the nuclear super powers.

According to participants, Senate Minority Leader Howard Baker (R-Tenn.) told Carter he would give a little more time before taking a strong stand publicly about the troops issue. Baker, who is a contender for his party's presidential nomination, is also reported to have said this is Carter's "test" by the Soviet Union and urged the president to rise to it.

Chairman Frank Church (D-Idaho) of the Senate Foreign Relations Committee, who three weeks ago today first publicly disclosed the administration's conclusion that a Soviet combat bridgade is in Cuba, is reported to have reiterated his position that SALT II cannot be approved without withdrawal of Soviet combat troops.

Church went on to say, according to participants in the meeting, that the Senate might attach understandings to SALT II saying that the treaty could go into effect only when Carter could certify that all Soviet combat troops had been removed, and saying that the United States retains the right to withdraw from the treaty if Soviet combat troops ever return to the Western Hemisphere.

Meeting reporters outside the White House after the unannounced early-morning meeting, Church called the current diplomatic negotiation "extremely vital" with "far-reaching effects upon our relationship with the Soviet Union."

He said a crucial issue is "the combat character of the [Soviet] force and the circumstances under which it is deployed."

Sen. Jacob Javits of New York, senior Republican on the Foreign Relations Committee, cautioned that the troops issue ought not "be blown up as a major national crisis" because everyone agrees that "this unit is no threat to the U.S. military." Javits added that "it does offend U.S. sensitivity" and that he is hopeful that the Soviets will realize this and take corrective action.