President Reagan yesterday abandoned negotiation for confrontation and challenged the House Democratic leadership to a head-on test of strength over his three-year tax-cut proposal.

As he did with his budget-cutting proposals, Reagan decided to push his program of a 5 percent cut Oct. 1 and 10 percent cuts on July 1, 1982, and July 1, 1983, with the aid of conservative Democrats in the face of opposition from the leadership of the Democratic-controlled House.

"If we don't have the votes, we'll get them," Reagan told reporters. White House chief of staff James A. Baker III predicted that the tax fight would be much closer than the budget contest, which turned into an easy Reagan victory over divided Democrats.

The president threw down his challenge in the Rose Garden with a small group of Republican and conservative Democratic members of Congress standing behind him. In a preview of the appeals he is likely to make to the public, Reagan called his election victory a "resounding message" from the American people.

"We believe that on economic recovery there can be no Republicans and no Democrats, only Americans," Reagan said.

After more than two weeks of negotiations with the House Democrats, Reagan threw down his gauntlet over two main remaining differences: a third-year cut and the share of the tax relief that would go to middle and lower income groups.

The Democrats had proposed a two-year cut of 5 percent this year and 10 percent next, and wanted to tilt the relief more heavily toward the middle and bottom of the income scales.

As he challenged the Democrats, the president also once more revised his proposals, to pick up votes and to save money. The most important new changes would cut estate taxes and significantly reduce the proposed depreciation write-offs for business.

Treasury Secretary Donald T. Regan was asked why the third-year tax cut for individuals was so important to the administration. "Generally because of planning," h replied. Regan said businesses and individuals will be able to plan investments better if they are certain what their three-year tax burden will be.

The Reagan proposal would cost the treasury $37.4 billion in 1982, $92.1 billion in 1983 and $144.5 billion in 1984, Regan said. Of the lost revenue, $224.8 billion would be from the relief for individuals and $49.2 billion from the relief for businesses.

Regan said the 25 percent cut over three years will "not do quite the job" of the 30 percent reduction Reagan made the centerpiece of his campaign "but will have generally the same effect."

In addition to the three-year cuts, the Reagan plan includes:

Faster tax write-offs for business investment in plant and equipment, but with a cap so that no business would get an outright subsidy for its investment.

A partial easing of the "marriage penalty" under which some two-income married couples pay more tax than they would if filing two single returns. In the first year, 5 percent of the income of the spouse earning less would be excluded up to $1,500. In the second year, this exclusion would be doubled.

A major change in death taxes that would permit a surviving spouse to inherit an unlimited amount without paying tax and would exempt estates under $600,000 from all taxes.

A one-year extension of the present exclusion of the first $200 of income from interest and dividends for an individual or $400 for a couple.

The total cost of the provisions for individuals other than the rate cuts would be $3 billion in the first year, $10 billion in the second year and $15 billion in 1984.

Regan said the administration still foresees a balanced budget in 1984.

The president said that Rep. Barber B. Conable (R-N.Y.), the senior Republican on the Ways and Means Committee, and Rep. Kent R. Hance (D-Tex.), a conservative member of Ways and Means, would introduce the president's bill.

Hance and his fellow conservatives who belong to the Conservative Democratic Forum are crucial to the outcome of the tax battle. On the budget, most of the 47 supported the president. The forum's chairman, Rep. Charles W. Stenholm (D-Tex.), said yesterday, however, that his group is split and that only 15 to 20 of them are firmly behind Reagan on taxes.

The president won the budget vote May 7 by 253 to 176.

"We don't think we're going to win by anything like the margin on the budget," chief of staff Baker said. He said that a number of the consevatives have made up their minds for a two-year tax cut and will stay with the Democratic leadership.

Rep. Ken Holland (D-S.C.) and Ed Jenkins (D-Ga.) said they were backing the Democratic plan announced Wednesday by Ways and Means Chairman Rep. Dan Rostenkowski (D-Ill.).

Until yesterday, however, Reagan had not personally joined the battle for his tax cut, but he proved during the budget debate to be an effective lobbyist. Members of Congress tropped to the White House for personal sessions with the president and walked away, in all cases, with a souvenir pair of cufflinks and, in most cases, having moved closer to the president's position.

Reagan yesterday called those legislators who support him part of "a new bipartisan coalition." He said that he White House and Congress are foregoing this coalition "on behalf of the American people."

His chief of staff also made clear that the administration, if blocked in the House, can exert pressure through the Senate. Baker said that in the Republican Senate the admiministration is in good shape, and that a Senate-passed bill much to its liking would still be brought to a Senate-House conference even if Reagan failed to get his way in the House.