Congress, throwing in the towel in its latest effort to revive the Gramm-Rudman-Hollings balanced-budget law, last night approved a short-term extension of the national debt limit that delays until September any further pressure to mandate automatic reductions in the federal deficit.

Despite a frenzied day of negotiations that drew House and Senate negotiators closer together in their search for a compromise on the deficit-reduction law, what one lawmaker described as "the siren song" of the annual congressional summer recess proved too enticing. Congress recessed last night and will return Sept. 9.

Following an earlier voice vote in the House, the Senate voted 51 to 39 to raise the debt limit to $2.35 trillion, giving the government sufficient borrowing authority to last until Sept. 23. That took the steam out of negotiations that congressional leaders had hoped would lead to a reinvigoration of the Gramm-Rudman-Hollings law, which was to have been part of extending the debt ceiling to May 1989.

By the time the Senate voted, congressional leaders had conceded that even if the negotiations yielded an agreement last night among conferees, there would be too few lawmakers in town to pass the measure on the House and Senate floors.

"We're losing our best opportunity to fix the Gramm-Rudman-Hollings law," Sen. Phil Gramm (R-Tex.), one of its coauthors, said before the Senate vote.

Others on Capitol Hill believe that Congress has not damaged its chances of reviving the balanced-budget law this session and with it the Democratic hope of forcing a budget and tax compromise with President Reagan.

"I think a substantial amount of progress has been made and will survive until after the recess," House Majority Leader Thomas S. Foley (D-Wash.) said.

Democratic leaders of the 100th Congress have been counting on a Gramm-Rudman-Hollings restoration as their best hope of getting Reagan to back off his opposition to a tax increase, which is a central element of the Democrats' $1 trillion fiscal 1988 budget. Repairing the automatic spending-cut mechanism in the balanced-budget law that was struck down last year by the Supreme Court, Democrats believed, would have given Reagan the choice of either accepting a tax increase or deep cuts in his defense buildup.

Under both the original Gramm-Rudman-Hollings law and the repair under consideration this week, the budget would have been cut across the board if Congress and the White House failed to agree on a package of deficit-reduction measures to meet the mandated annual deficit targets. Half of the automatic cuts would have come from Pentagon accounts and half from domestic programs, with some poverty and medical programs exempted.

Now, however, Congress will have only about two weeks after it returns from its recess in September to repair Gramm-Rudman-Hollings before the next debt-limit deadline. Congress also will face other pressures of unresolved legislation, as well as the Oct. 1 deadline for completing its spending and revenue bills to implement its budget.

The task, Gramm said, would mean "some heavy lifting for an athlete who hasn't run a race in a year and a half, and I'm talking about the U.S. Congress."

The denouement to this week's Gramm-Rudman-Hollings drama came after a day of furious negotiations between the House and Senate, with both sides exchanging a series of proposals to restore the automatic spending cut mechanism and relax the annual deficit goals to accomplish a balanced budget by fiscal 1993.

The task of reaching an agreement on revitalizing Gramm-Rudman-Hollings was complicated by the four-sided nature of the talks between Democrats and Republicans in both houses, all of whom had different ideas of how the law should be revised.

House members of the conference committee, for example, early yesterday voted to send a proposal to the Senate, but the plan was opposed by all 17 of the Republican House conferees in part because it called for few congressional budget reforms.

A subsequent counterproposal by the Senate was opposed by two key Republican conferees, Sens. Gramm and Pete V. Domenici (R-N.M.), who insisted it did not go far enough because it only mandated automatic budget reductions in the first three years of a six-year process.

Coupled with a shortage of time, that disagreement over the duration of the automatic spending-cut provision proved to be the back-breaker, despite broad agreement on new deficit targets, including a requirement to trim the deficit by $23 billion in fiscal 1988. Republican insistance on a six-year program of automatic cuts doomed the effort because the Democratic majority needed a significant measure of Republican help to pass any Gramm-Rudman-Hollings repair.

Partisan bickering about hidden agendas was a constant undercurrent during yesterday's frenetic negotiations: Democrats accused Republicans of trying to scuttle the talks in order to protect Reagan from tough decisions on the deficit and tax increases, and Republicans responded by charging that Democrats were interested only in humbling the president.