Top White House officials and congressional leaders have begun private talks aimed at avoiding an embarrassing confrontation over arms control next month during President Reagan's summit meeting with Soviet leader Mikhail Gorbachev.

White House chief of staff Howard H. Baker Jr., national security adviser Frank C. Carlucci and senior members of the House and Senate armed services committees tested the waters for compromise over lunch at the Capitol Wednesday and agreed to keep talking, according to congressional participants.

But there appear to be no guarantees that a compromise can be worked out on arms-control provisions that the House and Senate added to the defense authorization bill for this year over opposition and veto threats from the White House.

"We made some headway in understanding each other's positions," said Senate Armed Services Committee Chairman Sam Nunn (D-Ga.), who said he is "cautiously optimistic" that an accord can be reached.

"I got a sense that they {White House officials} want to work something out to avoid a fight during the negotiations . . . . There was no definite answer, but the whole discussion was 'talk and deal,' " said Rep. Les Aspin (D-Wis.), the House Armed Services Committee chairman.

Without a compromise, the White House and Democratic-controlled Congress could be locked in a veto showdown over the defense bill -- or a catchall spending bill that includes the same arms-control provisions -- just as Gorbachev is arriving for the summit, scheduled to start Dec. 7.

On the eve of his Iceland summit with Gorbachev last year, Reagan forced Congress into retreat with the argument that arms constraints by Congress would undermine the U.S. bargaining position. But when similar arguments were made during Senate consideration of arms provisions while Soviet Foreign Minister Eduard Shevardnadze was visiting last summer they had little effect.

In separate versions of the defense bill, the Senate and House have approved provisions that would restrict development and testing of spaced-based antimissile systems under the Strategic Defense Initiative (SDI) and force the administration to adhere to weapons limits in the unratified SALT II treaty.

The House bill would forbid the administration from departing from a traditional narrow interpretation of the 1972 Antiballistic Missile (ABM) Treaty in order to advance the SDI system, while the Senate measure would require congressional approval for any such departure.

The House bill also would ban all but the smallest nuclear weapons tests and continue a moratorium on testing of antisatellite weapons in space. These provisions were not approved by the Senate.

The SDI constraints are considered especially sensitive because disputes over space-based missile defenses figured prominently in U.S.-Soviet maneuvering leading up to the summit and are expected to loom large in broader talks that will start with the Washington meeting.

Nunn has indicated a possible compromise could involve a written presidential guarantee that any SDI testing would stay within confines of the narrow ABM interpretation.

A House-Senate conference has resolved most differences in the roughly $300 billion military authorization measure for fiscal 1988, which began Oct. 1, leaving the arms-control provisions as the only major issues to be worked out.

But Reagan has vowed to veto the bill if the arms constraints are included, and neither house is expected to be able to muster the two-thirds vote necessary to override a veto. However, if the bill is vetoed and the veto sustained, congressional leaders have said they will include the provisions in the defense appropriations bill for this year or, more likely, in an omnibus spending bill for the government that will include Pentagon funding.

Although Congress is set to adjourn Nov. 21, many lawmakers say it could still be in session, with defense-related issues unresolved, when Gorbachev arrives.

Meanwhile, 62 members of Congress wrote Reagan last week urging him to overrule the Pentagon and stop an upcoming test of a 12-warhead missile, called the D5, to be launched from a Trident II submarine. They said it would "complicate arms-control efforts and reduce the flexibility of our strategic forces."