With the White House confined to the sidelines, the Senate Budget Committee's Republican chairman and senior Democrat said yesterday that they were making "substantial progress" toward a bipartisan budget for next fiscal year that deviates sharply from President Reagan's tax and spending priorities.

Discussions reportedly centered on tax increases of up to $20 billion, along with sharp new limits on defense as well as domestic spending.

Although critical details were still to be resolved, both committee Chairman Pete V. Domenici (R-N.M.) and ranking Democrat Lawton Chiles (D-Fla.) said they were increasingly optimistic about the possibility of a bipartisan budget accord.

"We're not there yet, but we will be there," Domenici said. "We've got a good spirit . . . . We're getting closer," Chiles said.

In the past, Senate Republicans generally have tried to strike budget deals with the White House before going to the Democrats, a practice that has often led to partisan strife and stalemate.

But the frustration of last year's budget negotiations with the White House, coupled with new pressures from the Gramm-Rudman-Hollings budget-balancing law and other factors, have led to a reversal of the process this year.

Asked if he was attempting to negotiate with the White House in the same fashion as he was with Chiles, Domenici said, "No . . . not at this point." He acknowledged that this was a departure from previous practice but added, "Where it will end up I don't know."

Domenici invited Office of Management and Budget Director James C. Miller III to a caucus of committee Republicans yesterday afternoon but made it clear that it was not a negotiating session. Miller was invited, he said, to "hear where we are as a group on major issues, including defense."

While Domenici is pushing for more of a defense spending increase than most Democrats want, there is little support in either party for the increase of 8 percent, after accounting for inflation, that the administration has proposed. Moreover, Domenici has repeatedly warned that defense spending increases are impossible without raising taxes, which Reagan has opposed.

During a break in the closed-door meeting, Sen. Charles E. Grassley (R-Iowa) said Miller was sticking by the administration's defense figures. Republican senators who spoke out during the session told him "this just wasn't the real world," said Grassley, a critic of the Defense Department's spending plans. Democrats attributed the new bipartisanship on the committee, where Republicans hold 12 of 22 seats, to an increasing distance between Reagan and Senate Republicans on budget issues.

"If anything, it's the president's obstinacy and indications by the Republican leadership of the Senate that they're willing to move out on their own," said Sen. Howard M. Metzenbaum (D-Ohio), a member of the budget panel.

But Democrats, burned in the past by Reagan on the tax issue, are wary of signing onto a budget plan with tax increases unless a majority, perhaps more than a majority, of Republican members of the committee embrace it first.

Sen. Jim Sasser (D-Tenn.) said he thought seven Republican votes for a tax increase would be sufficient, but some Democrats were reported by their colleagues to be holding out for eight or nine.

At least four of the 12 Republicans on the budget panel have indicated they oppose a tax increase, although one or more have hinted that they might accept it in exchange for more defense spending. This, however, could make it more difficult to get the votes of Democrats and Grassley.

Domenici has spoken of tax increases totalling $12 billion to $20 billion for next year. Chiles said yesterday that he was pushing for the "upper range."

During a brief committee session yesterday, Domenici and others said the administration was unlikely to get the 11 percent foreign aid spending increase it wants, and members indicated uncertainty over how to budget for the space program.

In debate yesterday on a proposed constitutional amendment to require balanced budgets, the Senate rejected, 54 to 44, a move to require the president to submit a balanced budget.