One participant in the extended series of budget negotiations describes the process as "an exercise in improbability"; another views it as "a minuet in which no one wants to really dance."

Whatever they are called, the 12 meetings of congressional leaders and a White House negotiating team have revealed the difficulty of hammering out a genuine compromise between a president as fundamentally committed to tax reductions as Ronald Reagan is, and a House speaker as devoted to preserving entrenched social programs as is Thomas P. (Tip) O'Neill Jr.

The meetings -- the 12th of which ended inconclusively yesterday afternoon at the White House -- also have shown that Republican congressmen and the Reagan team led by White House chief of staff James A. Baker III have many differences among themselves despite the facade of unity that they have maintained outside the negotiating room.

The biggest difference has been over taxes. GOP supply-siders have long been worried that White House staff members are too willing to accept an income tax surcharge as part of a budget compromise, and this suspicion appears to have prevailed in the bargaining sessions.

"The surcharge is off the table," said Rep. Trent Lott (Miss.), the Republican House whip, after Friday's negotiation session. "The Democrats aren't pushing it, and we're not accepting it."

In the early stretches of the month-long negotiations, there were also differences on defense issues. Some Republicans resented what seemed to them an overly hard-line position by Defense Secretary Caspar W. Weinberger, which they thought had forced the White House team into a position too rigid even for military-minded congresssmen.

However, the word from the White House is that there has been some easing of the hard-line view. In the negotiations, defense cuts have on both sides of the table proved less of an obstacle than the tax issues.

The distrust among Republicans is mild in comparison with GOP suspicion of their Democratic counterparts in the negotiating process. The Republicans believe that the Democrats in the room with them are acting in good faith, but they are wary that any agreement they reach could be shattered on the House floor by rebellious Democrats, perhaps with the tacit approval of O'Neill.

"You could put up a nice package and be outmaneuvered procedurally," observes Senate Finance Committee Chairman Robert J. Dole (R-Kan.).

Lott said he is "frankly concerned that the provisions of the package could be undermined unless they are protected." For that reason, the Mississippian and other Republicans involved in the negotiations are talking about making a closed rule in the House part of the agreement. This would mean that the package would not be subject to amendment from the floor.

Procedural discussions preoccupied the negotiators in Friday's session, when it also became apparent that it would be impossible to come to detailed agreement on some of the toughest issues, such as a specific cap on entitlement programs or a decision as to the amount of revenues that would have to be raised by new taxes and the closing of tax loopholes.

"The good news is that we're likely to come up with an agreement," said one participant who has remained reasonably optimistic throughout the process. "The bad news is that it may not mean anything."

This participant's view was that a compromise -- considered likely to come this week, if it comes at all -- probably will be so general that the tough decisions will have to be fought out in Congress throughout the spring and summer, perhaps in still another round of leadership negotiations.

"These negotiations have a long history ahead of them," said one administration source.

Few doubt that the 17 men who have met together for approximately 40 hours of blunt but civil discussion could reach a compromise if the decision were left to them.

But the negotiators are reluctant to force their principals into a corner by taking an agreement to them that they are unlikely to approve. And Reagan, despite much talk about his willingness to "go the extra mile," has remained adamantly opposed to any diminution of the 10 percent income tax cuts scheduled to take effect on July 1 of this year and again in 1983.

Similarly, O'Neill has firmly opposed any cuts in Social Security cost-of-living increases, which White House negotiators have regarded as a likely part of a final package. Republican congressmen aren't eager for any such cuts in an election year either.

At the root of the difference is the ideological and personal gap between Reagan and O'Neill.

"The truth is that the president and the speaker, for all their geniality, don't really get along," said an administration source. "It's one of our biggest problems. Tip thinks that Reagan just cares for the rich and the Hollywood crowd. The president thinks that Tip is committed to all the old programs which got us into this mess."

What has kept the negotiations going is a shared concern, expressed by representatives of both sides, of what will happen if they collapse.

If there is no compromise, said White House policy development assistant Edwin L. Harper last week, interest rates and unemployment will remain at a high level and "everybody in Washington is going to be tarred with the notion that they had refused to take the next step."

Said Dole, at the end of the week's negotiations: "When I look at the alternatives, I'm not in any big rush to say 'Stop the meetings.' "

The premise of the negotiations, from the Republican point of view, has been that any package must show a declining trend line of budget deficits, starting with an '83 deficit that preferably should not exceed $100 billion. After that, the promised deficit is supposed to be reduced every year in what White House negotiators have referred to as a downward glide path.

"It doesn't have to be a nose dive but it's got to be a fairly steep glide," said Dole.

There are those in the White House who believe that, even without an agreement, the negotiations will prove politically beneficial to Reagan. They think that with the proper presentation he can emerge as a leader of both principle and flexibility who sought genuine compromise. There are those who believe that Reagan can exercise his demonstrated skills as "the great communicator" to affix a major portion of the blame for any budget failure on the Congress in general and the Democratic-controlled House in particular.

But there are other, more realistic views at work within the administration.

One of them, expressed by a long-term Reaganite, is that the president, "who will get most of the credit for a genuine compromise, will also get much of the blame for the failure to achieve one."

Another is that economic recovery, the major promise of Reagan's presidential campaign, is essential to restoration of his popularity. Those who hold this view recognize that a true budget compromise is the precondition for any such recovery.

"We oversold last year," said one administration source, "and there is not a lot of confidence out there, either in the country or on Wall Street. We need to bring the deficit down. And even if we do, this recovery is going to come bit by bit, without any dramatic change. Even if everything goes well, it will take us every month of the next three years to do it."