The Senate opened what promises to be a long, tough debate on next year's budget yesterday as its Republican leaders, over mounting opposition from within their party, vowed to push ahead with a controversial plan to squeeze $40 billion out of the Social Security system over the next three years.

If the kickoff speeches were any indication, there could be rough times for President Reagan's latest version of the budget, even in the GOP-controlled Senate, which gave the president's budget hardly a moment of trouble last year.

As both Majority Leader Howard H. Baker Jr. (R-Tenn.) and Budget Committee Chairman Pete V. Domenici (R-N.M.) held firm on the Social Security proposal, eight Republicans lined up against it, and Sen. Ernest F. Hollings (D-S.C.), ranking Budget Committee Democrat, characterized most of the budget's proposed savings as "phantom inducements suspended from doctored numbers."

"I'll tell you, it looks dreary," said Baker, speaking of chances that Congress can wrap up a budget by the start of its Memorial Day recess at the end of the month.

Disappointment was also expressed by Republicans in the strength of Reagan's commitment to the Social Security proposal, as he phrased it at a Thursday night news conference, during which he endorsed the plan but stopped short of insisting on its passage.

"I don't think his comments were terribly helpful," Domenici said. "He didn't read what I wrote," said Baker, who, with Domenici, had suggested a firmer statement.

But Domenici said he was optimistic that even the Social Security part of the budget would pass, possibly with some modifications, and a Republican aide said initial soundings indicated there would be enough Democratic defections to insure its adoption.

In brief remarks to a nearly empty chamber, only hours after a marathon debate over defense authorizations had ended shortly before dawn, Domenici defended the "now famous $40 billion," as he characterized it, in the strongest terms: "To leave it out is to deceive . . . to fail to tell the truth" about the huge retirement system's solvency problems. "I consider it to be truth-in-budgeting," he added.

But the eight GOP dissidents viewed it differently, arguing that Congress should not cut Social Security benefits to help reduce budget deficits or lock itself into any numbers before a special commission studying Social Security solvency reports back to Congress at the end of the year.

An amendment to strip the budget resolution of the $40 billion in Social Security savings, which many lawmakers say they believe would probably end up as benefit cuts, was introduced yesterday by Republican Sens. John H. Chafee (R.I.), David Durenberger (Minn.), Robert T. Stafford (Vt.), Richard G. Lugar (Ind.), William V. Roth Jr. (Del.), Lowell P. Weicker Jr. (Conn.), Paula Hawkins (Fla.) and Charles McC. Mathias Jr. (Md.). Most are Northeast and Midwest moderates who are up for reelection this fall.

Chafee and Durenberger are also co-sponsoring a separate amendment to expand the committee's proposed tax increase while cutting more in defense and less in education and health programs. Democratic senators plan more far-reaching amendments, including ones to help reduce future deficits by scuttling the 1983 tax cut.

The committee's proposed budget of $779.1 billion for fiscal 1983 includes $20 billion in tax increases, $5 billion in defense cuts, $7 billion in entitlement program cuts and a freeze on domestic appropriations as well as federal pay and pensions.

It also includes $6 billion of the $40 billion in unspecified Social Security savings that would be made through 1985.