The surtax, a revenue-raising mechanism last used to help finance the Vietnam war, has been moving on and off center stage in the budget negotiations between the Reagan administration and Congress.

A complex idea in itself--it amounts to a tax on a tax--a surtax is seen by some negotiators as a vehicle to bring two major adversaries, President Reagan and House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.), into the negotiating process.

On this theory, a surtax, particularly one directed toward high income taxpayers, could meet O'Neill's objection that the third year of the tax cut needs modification or repeal. At the same time, it would allow the president to claim that his tax program had not been gutted by the budget compromise.

But the proposal has already provoked the anger of at least nine conservative senators, led by Willianm V. Roth Jr. (R-Del.), who wrote to Reagan:

"This proposal would indirectly subvert the recently enacted individual tax rate reductions. We are firmly and unequivocally opposed . . . to any general tax increase on the working men and women of America."

As if this does not make the political debate surrounding a surtax complex enough, Senate Finance Committee Chairman Robert J. Dole (R-Kan.) recently entered the fray with the argument that a surtax, even if geared to persons in high tax brackets, would not respond to what he considers to be the most serious equity issue.

Dole argues that the major tax fairness issue facing Congress is the evasion of tax liability by profitable firms and wealthy individuals using legal tax preferences and loopholes. A surtax on these taxpayers would be "a surtax on zero" according to Dole, who would much prefer to press for a broad "minimum" tax on rich people and corporations.

The Vietnam war surtax was highly unpopular, both among those who had to pay it and among employers who had to figure out how to withhold it, according to congressional sources. This history will not make it any easier to include in a budget package.

The war surtax, passed in 1968, worked in a significantly different way from the general outline of proposals now under consideration by the White House and Congress.

In contrast to the 1968 law, which set the tax at 10 percent, lawmakers are now looking at a 4 percent surtax. Although subject to considerable modification, the basic outline would likely be as follows.

In a key political decision, Congress and the administration would have to decide how much money they wanted to raise from the surtax and which income groups should pay. The two issues are locked together because the more income groups that are covered, the larger the amount of revenue raised.

A second political decision would be whether to apply the surtax to the corporate income tax. At 4 percent, the addition of corporations would add about $2 billion annually in revenues.

One proposal floating in the negotiations would be to limit the surtax to average-sized families with incomes in excess of $40,000. This would raise about $4 billion annually, and would allow politicians to claim they were not increasing the tax burden of most working American men and women.

Using a "typical" family of four with one person working at a salary of $40,000, the general calculation is that, after deductions and personal exemptions, the family filing a joint return would have a "taxable" income of $30,200.

This translates into a federal tax liability of $5,124 in 1983. The $5,124 figure is the key number for all four-person, one-earner families under this scheme. For all taxpayers in this category whose federal tax liability is below $5,124, the surtax would not apply.

For a taxpayer in a one-earner, four-person familiy making $100,000, for example, the typical tax would be $23,230 under current law. To calculate the surtax, the taxpayer would first subtract $5,124 (the amount exempted from the charge) from $23,230 to get $18,106.

This taxpayer would then multiply the $18,106 by 4 percent--0.04--to get $724. This $724 would be the surtax, and would raise total tax liability to $23,954.

For someone making $50,000 in this family size, the typical tax liability would be $7,569 in 1983, meaning that the surtax would apply to the difference between $5,124 and $7,569, or to $2,445. The surtax for that taxpayer would be 4 percent of $2,445, or $98.

Under this scheme, the surtax would apply at far lower levels of income for single taxpayers who have a different rate structure. For the single taxpayer, the surtax would start to become applicable when income exceeded about $29,000.

Another proposal under consideration would broaden the surtax by making it apply to families making $25,000 or more, instead of $40,000. This would raise about $7 billion, but could make Congress and the administration subject to criticism that a general tax increase had been applied to many working and lower middle-class families.