Congressional negotiators failed again last night to achieve a compromise on legislation that would both resuscitate the Gramm-Rudman-Hollings balanced-budget law and extend the federal debt limit.

House negotiators reached a tentative accord among themselves on a revised timetable for bringing the federal budget into balance through a new mechanism that would trigger automatic, across-the-board spending cuts if Congress and the White House failed to reach the deficit targets.

However, a final agreement among the House negotiators hinged on parallel talks producing sufficient changes in the congressional budget process itself to satisfy Republican lawmakers.

"We are in general agreement, but we reserve the right to see the results of budget reform," said House Minority Whip Trent Lott (R-Miss.). "If we make progress on budget reforms, we may have something."

By the time negotiators broke off their talks late last night, that agreement was still proving elusive and Senate negotiators were divided among themselves over how long the new automatic spending-cut provision should last.

Sen. Lloyd Bentsen (D-Tex.) described the talks as "painful and slow." Sen. Bob Packwood (R-Ore.) predicted that "with a long day tomorrow {Friday} we can reach agreement on a conference package but not necessarily with unanimous agreement."

Meanwhile, House and Senate leaders said they are determined to get an agreement on reinvigorating the Gramm-Rudman-Hollings law in time for Congress to recess for its August holiday today or Saturday. Adding to the pressure was a fear among House leaders that the Senate would not go along with yet another short-term extension of the federal debt limit, this time until September.

If a Gramm-Rudman-Hollings agreement is achieved, it will be attached to legislation that allows the federal government to borrow up to $2.8 trillion, which would give the Treasury sufficient borrowing authority through May 1989. The current debt ceiling of $2.3 trillion was scheduled to revert to $2.1 trillion at midnight last night. Without an extension, the government would run out of cash and default on its security obligations beginning Aug. 17.

Negotiations between the House and Senate over new deficit targets and a revision of the automatic-spending-cut mechanism that was struck down last year by the Supreme Court bogged down Wednesday night. At that time, Senate conferees rejected a House demand that the spending-reduction provision be in effect for just two years -- that is, until the end of the Reagan administration.

But yesterday afternoon, Democrats and Republicans in the House agreed in principle to an amendment that would have the "trigger" that would require automatic spending cuts in place for three years and would require the government to have a balanced budget in fiscal 1993, two years later than the timetable in the existing Gramm-Rudman-Hollings law.

In addition, the changes would mandate a $23 billion reduction in the federal deficit -- currently estimated at $182 billion -- in fiscal 1988, which begins Oct. 1. That reduction would come through automatic across-the-board spending cuts if Congress and the administration cannot agree on a package of budget cuts and revenue increases to accomplish the goal. For fiscal 1989, the amendment calls for a maximum further reduction of $36 billion.

Under the tentative agreement, President Reagan would have some flexibility over the next two years in determining which individual defense programs would be affected by the mandatory spending reductions. As in the original budget law, half the automatic spending reductions would come from the Pentagon and half from domestic programs.

If the agreement holds, it would require about $14 billion less in deficit reduction than called for in the $1 trillion budget adopted by Congress. It would also significantly reduce the $19.3 billion tax increase that is the controversial centerpiece of the budget and that Reagan has vowed to veto.

A key unresolved question is whether the new deficit timetable will be acceptable to the Senate, where Republicans hold considerably more power than in the House.

Although Packwood said he could accept a three-year automatic trigger, two other key senators in the conference -- Phil Gramm (R-Tex.) and Pete V. Domenici (R-N.M.) were holding out for an automatic mechanism that would remain in place until the federal budget is balanced.

At a morning news conference yesterday, Gramm said he is adamant that any provision for automatic spending cuts extend beyond the life of the Reagan administration. By trying to limit it to two years, Gramm said, "Democrats are just trying for force Ronald Reagan to accept a tax increase. Unless we have a full five-year program {that includes automatic cuts}, I am not going to support imposing on Ronald Reagan restrictions we won't impose on the next president."

Nonetheless, a Senate aide familiar with the negotiations said "the probability is increasing" for an agreement that can pass both houses. But the aide predicted that Senate Republicans will insist on retaining the automatic-spending-cut mechanism for five years and on changing the budget process.