Efforts to boost spending for domestic social programs are likely to fail when the Senate takes up the 1981 budget next week, Senate Majority Leader Robert C. Byrd (D-W.Va.) predicted yesterday.

The West Virginia Democrat defended the $612.9 billion committee proposal as "a balanced budget approach" that the full Senate will endorse. "In the main, the budget will emerge pretty much in the present form," Byrd said.

Major amendments to be offered by Sens. Alan Cranston (D-Calif.) and Gaylord Nelson (D-Wis.) and others would reduce the defense portion of the proposal and increase spending for social needs. Efforts in that direction failed during budget committee meetings last month, and the defense proposal now stands at $155.7 billion, $5.2 million more than President Carter recommended.

Byrd told his regular Saturday news conference that he had not yet decided whether to support Cranston's effort, a position generally thought to indicate his belief that it will not succeed.

The budget-balancing act could be endangered by the rising unemployment rate, which will cut tax revenues and increase welfare outlays, Byrd acknowledged, but he said it was too early to plan any change of course because of that problem. "The worst thing we can do is jump ship too soon," he said.

As the budget showdown draws near, Byrd's mail has shifted from rousing calls for a balanced tally to pleas that particular programs be saved from the ax, he said. Amendments along those lines could take the budget debate past next week.

Sen. Edmund S. Muskie (D-Maine) supported less defense and more domestic spending in his capacity as Senate Budget Committee chairman, but he has a history of backing the full committee verdict and will probably support the existing package on the floor next week, Byrd predicted. He praised Muskie's nomination as secretary of state, saying the senator's quick temper "may be an asset in that job."

Muskie "will be able to hold his own in upcoming meetings with Soviet Foreign Minister Andrei A. Gromyko, and should use the occasion to "express U.S. indignation face to face about the atrocities in Afghanistan" by Soviet troops, Byrd said. "It ought to be done forcefully and he will do it forcefully," he added.

On other foreign policy matters, Byrd said he admired the courage of thousands of Cuban refugees pouring into Florida for "voting with their feet . . . giving a black eye to Cuban Premier Fidel Castro and his totalitarian regime."

However, he said, other nations should join the United States in offering new homes to the exiles since domestic unemployment and economic problems mean "we simply cannot be expected to take the full load." This country has no obligation to provide refuge for common criminals, he added, although newly released political prisoners should be made welcome.