Senate Republicans edged closer to President Reagan's positions on defense and taxes, but continued to resist most of his proposals for domestic spending cuts as they struggled toward a compromise budget with a projected deficit of nearly $200 billion for next year.

As the Senate began a week of debate on the fiscal 1984 budget, the 54-member Republican majority failed in two closed-door meetings yesterday to nail down a party-line position on the tax-and-spending blueprint the Budget Committee approved last month.

But Budget Committee Chairman Pete V. Domenici (R-N.M.), who outlined the draft of a compromise to his colleagues yesterday morning, said after a second meeting last night that they were making "significant progress," although they still lacked the 51 votes needed to pass a package of their own.

There will be "some lack of unanimity early on" as the Senate proceeds with the budget, said Finance Committee Chairman Robert J. Dole (R-Kan.), "but I think eventually we'll pass it."

Several GOP senators, from both the left and right, indicated they could not support the budget revision package. Some also questioned whether modifications could pick up the necessary votes. "I'd say it's either going to be Domenici's plan or nothing," said Sen. Charles E. Grassley (R-Iowa), who objects to the proposed increase in defense spending.

To get a budget out of the severely split Budget Committee, Domenici and other Republicans had to make major concessions to the Democrats, such as agreeing to $30 billion in new taxes next year. They can shake free of these if 51 Republicans can agree on an alternative.

"It goes a little against the grain when they go for a budget that would get 51 votes and shut our folks out," Minority Leader Robert C. Byrd (D-W.Va.) complained after he was informed of the GOP plan.

The proposed compromise, which Domenici drafted over the weekend after consultations with disparate party factions, would grant Reagan's request for only minimal tax increases over the next two years. Bigger tax increases for future years would be suggested but not required, in line with Reagan's proposal for $150 billion in "standby" taxes for fiscal 1986, 1987 and 1988.

The plan would split the difference on defense for next year by providing an after-inflation increase of 7.5 percent, halfway between Reagan's proposed 10 percent and the Budget Committee's 5 percent. Increases of 5 percent a year would be provided thereafter. These, like the Budget Committee's plan, would give Reagan nearly all he wanted for defense, but over a longer time.

On domestic spending, however, Reagan would have to swallow $11 billion more than he wants, which is what the Budget Committee had proposed in accordance with pressure from the more liberal Republicans and the committee's Democratic minority.

Largely because the proposed tax agreement would drop all but $2.3 billion of the $30 billion in new fiscal 1984 taxes that the Budget Committee proposed, the plan's deficit would exceed all previously proposed red-ink levels for next year.

These included $189 billion in Reagan's original budget, $174 billion in the House-approved budget and $163 billion recommended by the Senate Budget Committee.

"It's a great misfortune," said Majority Leader Howard H. Baker Jr. (R-Tenn.), speaking of the deficit, which, along with taxes and defense, was reportedly a stumbling block in the morning meeting. "But this is as good as we can do at the moment," Baker added.

Not so, said Sen. Lawton Chiles (D-Fla.), ranking Democrat on the Budget Committee, in his opening remarks in the budget debate.

"I can tell you right now how we can bring those deficits down," Chiles said. "We can do it by adopting the budget resolution reported by the Senate Budget Committee."

In defending the committee's draft, Chiles said that the proposed tax increase, which could largely be accomplished by repealing the 10 percent income tax cut scheduled for July, is necessary to continue economic recovery. "Nothing will kill the recovery quicker than interest rates going up, and nothing will keep those rates higher than huge federal deficits," he argued.

But any Republican agreement would apparently collapse under conservative objections if it mandated major tax increases. Domenici's plan skirted this by requiring only about $7 billion in tax increases through 1985 and suggesting but not requiring revenue increases for future years.

Sens. William L. Armstrong (R-Colo.) and Orrin G. Hatch (R-Utah), who had voiced strong objections to tax increases, indicated that they could support Domenici's proposal, at least as a last resort.