The Senate yesterday defeated a Democratic proposal and refused to take up a Republican proposal to put the bite back into the Gramm-Rudman-Hollings balanced-budget law. Negotiators are expected to return to the bargaining table to fashion a compromise in time to prevent an unprecedented U.S. default on its security obligations.

The votes constitute a setback for Democratic leaders of the 100th Congress who view restoration of the budget law's automatic spending cut mechanism as their best hope of forging a spending and tax compromise with the Reagan administration.

Both the Democratic and Republican proposals, which also would have stretched out the goal of a balanced budget by one year to fiscal 1992, were offered as amendments to must-pass legislation to raise the national borrowing limit to $2.6 trillion through September 1988.

Last Friday, the debt ceiling fell automatically from $2.3 trillion to $2.1 trillion, and the U.S. Treasury has warned that failure to raise the limit by Aug. 1 would force a U.S. default, with devastating repercussions on the marketplace.

The Democratic proposal offered by Senate Budget Committee Chairman Lawton Chiles Jr. (D-Fla.) would have eased the yearly deficit targets under Gramm-Rudman-Hollings by requiring a maximum annual deficit reduction of $36 billion. It was defeated 71 to 25. Under the plan, Congress and the White House would have had to either reduce the fiscal 1988 deficit by $36 billion, or lower it to a level of $150 billion, whichever amount required the smaller reduction.

A Republican alternative offered by Sens. Pete V. Domenici (R-N.M.) and Phil Gramm (R-Tex.) that would have established a new set of fixed yearly targets, was blocked from coming to a vote on a procedural motion. Republicans fell 13 votes short of the 60 votes necessary to bring the plan to a vote.

Without a change in the deficit target, the budget adopted earlier by Congress would fall more than $35 billion short of meeting the existing fiscal 1988 goal of $108 billion.

Both the Democratic and Republican plans would have replaced the Gramm-Rudman-Hollings automatic spending cut mechanism that was overturned by the Supreme Court last year. The original spending cut mechanism and the two plans rejected last night called for across-the-board budget reductions if Congress and the White House do not meet their deficit goals. But the versions rejected last night were designed to pass constitutional muster. They differed over the executive branch's role in triggering the across-the-board cuts and its flexibility in carrying them out.

Each would have retained a requirement that the automatic spending cuts be equally apportioned among defense and domestic programs, but the Republican alternative would have given the administration some leeway in deciding which military programs bore the brunt of the cuts.

Democrats who control the 100th Congress, many of whom opposed the Gramm-Rudman-Hollings law when it was introduced in 1985, now view restoration of the automatic spending cut trigger as the central lever in getting President Reagan to accept their $1 trillion budget for fiscal 1988. Reagan has repeatedly promised to veto the $19.3 billion tax increase called for in the Democratic spending plan.

Democrats hope that Reagan might relent and support the tax increase, or at least negotiate a new deficit reduction package, if he is faced with automatic budget cuts that would slash deeply into his prized defense buildup.

The Senate's vote occurred amidst increasing financial and time pressures, as the Treasury Department had to dip into cash reserves to refund $13.7 billion in Treasury bills that matured yesterday. Treasury officials had already postponed last Monday's weekly auction of Tresaury bills to ease demands on the nation's credit.

With the debt limit lowered to $2.1 trillion, and less than $30 billion in cash on hand, Treasury officials said the United States will be unable to redeem $10.2 billion in two-year notes that mature on July 31. Usually maturing notes are repaid out of the proceeds from new securities issues.

Though Democrats and Republicans spent several days searching for a bipartisan plan to beef up Gramm-Rudman-Hollings, the talks fell apart Wednesday.

Domenici and Gramm said last night that their position was strengthened by rejection of the Democratic plan. "If {Democrats} want to negotiate, we'll talk, but clearly we start with our amendment, not theirs," Domenici said.