House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) yesterday accused President Reagan of making "vicious remarks" reminiscent of the anti-communism campaigns of the "early 1950s" in assailing a House Democratic alternative budget last week.

The first budget resolution for fiscal 1984 is scheduled to go to the House floor later this week in the most important test of strength so far this year between Reagan and the Democrats.

In denouncing the Democrats for seeking to cut his proposed defense spending increase by more than half, Reagan said last Friday that "nothing could bring greater joy to the Kremlin than seeing the United States abandon its defense rebuilding program after barely one year."

"It's regrettable that the president stoops to make statements like that," O'Neill said in likening Reagan's comment to some that were made in the era 30 years ago, when Sen. Joseph R. McCarthy (R-Wis.) held sway. But when asked if he was accusing Reagan of "McCarthyism," by which the era came to be known, O'Neill said he was only speaking of a time when "a lot of people saw communists under every bed."

O'Neill attacked the president as well for seeking delays and obstructing the congressional budget process that the president exploited so successfully over the past two years.

"The president reminds me of the little rich kid who, seeing his side begin to lose, says, 'It's my ball. I'm going home,' " the speaker said.

While lashing back at the president in some of the strongest terms he has used since the start of the Reagan administration, O'Neill also tacitly conceded that Reagan's broadside on Friday may have made some inroads among conservative Democrats.

Within hours, O'Neill's fears were confirmed as the 38-member Conservative Democratic Forum unveiled an alternative of its own and urged--without success--that the Rules Committee let the House vote on it when the first budget resolution comes up for debate today or Wednesday.

The "Boll-Weevil" alternative calls for a defense spending increase of 7 percent after accounting for inflation, which is halfway between the 10 percent proposed by Reagan and the 4 percent recommended by the Budget Committee.

It also would trim the committee's proposed $30 billion tax increase to $10 billion and knock out most of the domestic spending increases recommended by the Budget Committee, allowing step-ups only for education and health.

Some Democratic conservatives, including Rep. Charles W. Stenholm (D-Tex.), said they would vote against the Democratic budget resolution unless it was changed to reflect their views, raising the possibility that Reagan could win a conservative coalition victory again this year as he did in 1981 and 1982.

But others said the Boll-Weevil faction was split. At least a bare majority of the group is leaning toward the Democratic budget, said Deputy Majority Whip Bill Alexander (D-Ark.).

Even without Republican support, House Democrats--who picked up 26 seats in last year's elections--figure they can lose 48 of their own votes and still win.

"I think we're going to beat him Reagan , especially if he makes it a partisan issue," Alexander said. "He helps us, rather than hurts us."

In closing out the conservative alternative, however, the leadership-controlled Rules Committee also shut out substitutes sought by the Congressional Black Caucus and by Rep. William E. Dannemeyer (R-Calif.), while allowing Republicans as a whole to offer an alternative even though the party did not ask to. The Republicans' strategy has been to defeat the Democratic budget resolution rather than push for Reagan's or offer an alternative of their own.

A fight, possibly today, is expected over the rule governing debate of the budget resolution, and the Rules Committee may have shut out enough groups to jeopardize passage of its proposal and thereby throw the budget open to a whole string of amendments.

In attacking Reagan's "Kremlin comment," O'Neill noted that the Democratic budget resolution, while trimming Reagan's request, still would increase military spending by $45 billion next year and a total $217 billion through fiscal 1986.

As for Reagan's charge that the spending levels in the proposed $863.5 billion Democratic budget resolution amount to a "dagger aimed straight at the heart of America's rebuilding program," O'Neill said the Democratic budget would reduce the level of government outlays to below those of the administration by fiscal 1986.

O'Neill also chided Reagan for urging the Senate Budget Committee to delay action on its version of the budget resolution when it appeared the committee would defy him and cut his defense increase to 5 or 6 percent.

"The same man who complained of congressional delays has now asked the Senate Budget Committee to forget its timetable," O'Neill said.

He added, "The same man who once complained that Democrats lacked alternatives now urges House Republicans to forget his own budget and simply obstruct."

House Minority Leader Robert H. Michel (R-Ill.) said yesterday that he told Reagan that Republicans would probably not be able to defeat the Democrats' budget.