The Senate, over protests from liberals and grumbles from conservatives, last night approved a $784 billion budget for the next fiscal year that would raise taxes, cut defense and domestic spending and still leave a federal deficit of nearly $116 billion.

The vote was 49 to 43, largely along party lines.

The final round of budget votes in the Republican-controlled Senate came as the Democratic House began debate on half a dozen budget plans and nearly 70 amendments, with votes not scheduled until next week and the outcome very much in doubt.

While the White House has indicated that the Senate budget is acceptable to President Reagan, the House plan that comes closest to meeting his priorities faces serious challenges that have been drawn up by the House Budget Committee and by a moderate bipartisan coalition.

The congressional action follows more than three months of intensive efforts by members of both parties to devise an alternative to Reagan's fiscal 1983 budget, which was virtually dead on arrival when he submitted it to Congress last February.

Even after the two houses approve their own budget resolutions, there is still a long way to go: a House-Senate conference to resolve differences and legislation to carry out cost-cutting mandates in the budget, without which the 1983 deficit could exceed $180 billion, according to congressional budget experts.

In the Senate, Republican leaders succeeded in blocking most Democratic proposals to restore spending for programs ranging from infant care to veterans' benefits, with the major exception being $400 million for railroad retirees. The restoration would total $1.7 billion by 1985.

The railroad retirees proposal, which would guarantee full cost-of-living increases for nearly 1 million pensioners throughout the country, was approved, 57 to 40, after GOP leaders failed in several almost desperate attempts to block or modify it.

So fragile was the consensus behind the budget proposal that Republican leaders feared any major restorations of funds would shatter it by prompting conservative Republicans, already unhappy about tax increases and the big deficit, to join Democrats in rejecting the measure for different reasons.

But the leaders held the line on the next major proposal, which would have extended unemployment benefits for an extra 13 weeks--to 52 weeks overall--in states of high unemployment, at a cost of $337 million next year and more than $1 billion over three years.

It was defeated, 52 to 42, despite Democrats' protests that the recession has shredded Reagan's "safety net" for millions of American workers and Republicans' acknowledgement that the unemployment problem is acute.

On the final vote, conservative Democrats Howell Heflin (Ala.), John C. Stennis (Miss.) and Edward Zorinsky (Neb.) joined all but two Republicans in supporting the resolution. The two Republicans represented the philosophical extremes of the GOP in the Senate: Lowell P. Weicker Jr. (Conn.) on the left and Jesse Helms (N.C.) on the right.

Charles McC. Mathias Jr. (R-Md.) and John W. Warner (R-Va.) voted for the budget resolution, while Harry F. Byrd Jr. (Ind.-Va.) and Paul S. Sarbanes (D-Md.) voted against.

Shortly before the final vote and despite Senate leaders' praise for the proposal, the Senate voted, 70 to 21, against a substitute advanced by Sen. J. Bennett Johnston (D-La.) that would have produced a balanced budget by 1985 by raising taxes and cutting defense spending more than Reagan and the Budget Committee wanted.

It also would have placed new limits on future cost-of-living increases for Social Security and other benefit entitlement programs. In contrast, the committee's proposal would leave the budget out of balance by $65 billion in fiscal 1985.

The Senate also rejected, 56 to 34, a proposal to the to repeal future adjustments, or "indexing," of tax rates to account for inflation.

After the vote, Senate Budget Committee Chairman Pete V. Domenici (R-N.M.) said that if the House approves a similar budget, "we can go out in the marketplace and convince the money market people . . . that we have begun to dramatically reduce deficits" while preserving Reagan's tax cuts and defense buildup.

Domenici noted that the projected defcit by 1985 had been reduced from a possible $233 billion to $65 billion, and said, "That's a feat."

The resolution was drafted by the Senate Budget Committee after it unanimously rejected Reagan's original budget which, even with its own spending cuts and tax increases, contained a deficit calculated by the Congressional Budget Office at more than $130 billion for next year, with substantially higher revenue-spending gaps for future years.

The Budget Committee's substitute carried Reagan's blessing although it departed substantially from his earlier proposal, including higher taxes and deeper cuts than he wanted in his big defense buildup.

The budget includes $107 billion in tax and user fee increases by fiscal 1985, compared with Reagan's proposal for less than $40 billion over the same period, although it does not recommend deferral or modification of Reagan's controversial individual income tax cuts.

It also includes $22 billion less in defense spending over three years than the president wanted.

In place of many of Reagan's controversial spending cuts, it includes a three-year freeze on domestic appropriations and a one-year freeze on federal workers' pay and pensions, which would be followed by 4 percent raises in future years.

The budget would cut $26 billion from major benefit entitlement programs, such as Medicare and Medicaid, although one of its most controversial and ambitious provisions--$40 billion in savings from Social Security over three years--had to be shelved in the face of opposition from many Republicans and Democrats.

In a further effort to nail down a majority of the Senate, where Republicans control 54 of 100 seats, GOP leaders had to agree to additional spending add-backs of $19 billion by 1985, along with $6.2 billion in additional tax increases over the same period.

In effect, this partially protected several popular programs, including space and science, housing, Medicare, veterans' health and guaranteed college student loans.

These concessions appeased Republican moderates but caused anguish among conservatives, who said they might vote against the budget if the deficit--which the Senate once hoped to keep under $100 billion--crept up any further.

This was one of the main reasons that Majority Leader Howard H. Baker Jr. (R-Tenn.) and Budget Committee Chairman Domenici fought so hard to fend off spending restorations, prompting Democrats to claim they were being "gagged" and dismissed out of hand for partisan reasons.

In a particularly testy debate yesterday, Minority Leader Robert C. Byrd (D-W.Va.) accused Republicans of automatically rejecting every Democratic proposal, even to the point of tabling amendments without allowing an up-or-down vote.

Byrd grew even angrier when Sen. Alan K. Simpson (R-Wyo.) responded that Byrd, when the Democrats were in power, used to run a "railroad train . . . through this chamber in extraordinary ways, usually at night, late at night."

"You're living through exactly what we lived through," added Sen. Barry Goldwater (R-Ariz.). Finally Byrd, with a dozen or more Democratic amendments backed up on the agenda, had to cancel a party "retreat" that had been planned for the weekend in West Virginia.

The Senate budget resolution of $784 billion for fiscal 1983 compares with a budget calculated by the Senate at $740 billion for the current fiscal year, a small increase reflecting less than the inflation rate for the period. The projected 1983 deficit of nearly $116 billion is roughly similar to the $117.7 billion deficit expected for the current fiscal year, ending Sept. 30.

Technically, it is a first budget resolution for the year, setting spending and revenue targets that will be followed later by binding ceilings.

But, like Reagan's budget last year, it enforces its targets with "reconciliation" instructions to legislative committees ordering them to come up with actual tax increases and spending cuts. The deadline on the Senate budget is June 18.