A proposal to force sweeping deficit reductions leading to a balanced budget within five years attracted broad support in the Senate yesterday. Several members predicted that legislation raising the federal debt ceiling to $2 trillion cannot pass unless it includes this new plan.

About 40 senators, including several Democrats, are cosponsors of a Republican-drafted amendment to reduce the deficit from $180 billion to zero by threatening mandatory across-the-board spending cuts if annual targets for deficit reduction are not met.

They plan to push it as an amendment to the debt measure, which Treasury Secretary James A. Baker III has said must be passed by Monday so the government can keep borrowing to pay its bills.

By Monday, the government's cash balances will be "virtually exhausted . . . and the situation will deteriorate sharply thereafter," said Baker in a letter to Senate Majority Leader Robert J. Dole (R-Kan.).

The surge in support for the deficit-reduction measure, drafted by Sens. Warren B. Rudman (R-N.H.) and Phil Gramm (R-Tex.), prompted moves by the Democratic leadership in both houses to come up with counterproposals.

In the Senate, Minority Leader Robert C. Byrd (D-W.Va.) was working with Sen. Lawton Chiles (D-Fla.), ranking Democrat on the Budget Committee, on a plan that included tax increases as well as spending cuts.

In the Democratic-run House, Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) and other party leaders were exploring strategy for dealing with the debt bill if, as they expect, the Senate sends it back to the House with a deficit-reduction rider.

"Almost any balanced-budget proposal could pass the House" unless a more attractive option is proposed, said a House Democratic aide, who said a task force is likely to be appointed to draft an alternative deficit-cutting plan.

Under the Gramm-Rudman proposal, the president and Congress would be required to keep their budgets within fixed limits, eliminating the deficit by fiscal 1991. If the limit is exceeded by more than 5 percent, the president would be required to cut equally from entitlement programs and discretionary spending, including defense.

The threat of such cuts is expected to move Congress to make deficit reductions, possibly including tax increases as well as spending cuts. Some backers of the proposal indicated that it would improve chances for higher tax, although Gramm, a tax-increase foe, said the proposition could be argued either way.

At a news conference to demonstrate support for their proposal, Gramm and Rudman said they think that it has enough votes to pass the Senate and contended that the debt measure cannot pass without it, a view shared by many.

With mounting public concern over deficits, reinforced by the symbolism of a $2 trillion debt only five years after indebtedness passed $1 trillion, many lawmakers are reluctant to raise the ceiling without a demonstration of intent to control deficits.

For instance, Sen. Christopher J. Dodd (D-Conn.), a liberal who joined in cosponsoring the Gramm-Rudman proposal, said he would not vote to raise the debt limit without deficit reductions, even though he always supported debt-ceiling increases in the past.

As a sweetener to lure Democrats, especially in the House, the drafters exempted Social Security from mandatory across-the-board cuts that would occur if annual deficit-reduction targets are not met. That move paid quick dividends. Sen. John F. Kerry (D-Mass.) entered the room as the news conference broke up, saying he supports the measure now that he has been assured Social Security would not be cut.

Rudman and Sen. David L. Boren (D-Okla.) indicated that they and others would filibuster, if necessary, to assure passage of the proposal as part of the debt measure. At least, said Gramm, they want to force the House to vote on the issue.

President Reagan has indicated that he likes the Gramm-Rudman approach but has called for passage of the debt legislation without amendments. Senate Republican leaders, caught between conflicting pressures from their troops and the White House, planned to meet with fellow GOP senators today to decide on strategy for dealing with the bill.