Threatening that government checks could start bouncing today, the Reagan administration yesterday increased the pressure on Congress to pass an extension of the debt ceiling that includes mandatory balanced-budget provisions.

The administration warning came as Senate leaders of both parties, faced with a stalemate in the chamber, met privately to negotiate a compromise that could pave the way for at least temporary extension of the government's borrowing power.

In what has increasingly become a high-stakes game of chicken, Republicans have been insisting on including in the debt measure a far-reaching new set of budget rules that would require a balanced budget by 1991.

Senate Democrats, with the support of their House colleagues, have been holding off action with threats of a filibuster. Republicans have not had enough votes to break a filibuster, even though they have a majority for eventual passage of the budget plan.

The Democrats are seeking a temporary debt extension. The Republicans have so far blocked it, hoping to use the pressure of the debt-ceiling deadline as leverage to pass the new budget rules.

The threat of an unprecedented government default on its obligations has been raised daily in the current debate. But the deadline was vague, subject to the flow of government funds and judgmental decisions by Treasury officials. See related story, Page A17.

However, a letter yesterday from Deputy Treasury Secretary Richard Darman upped the stakes by threatening a deadline. Some escape hatches, such as emergency borrowing through the Federal Financing Bank, would remain if the government declares itself out of cash.

It appeared that any immediate effect of the government's losing its ability to borrow would be limited to government contractors, officials said. Social Security checks for this month were mailed last week, and most federal workers were paid yesterday. Any checks that had not been cashed by the time the Treasury Department declared the government out of money could be in at least temporary jeopardy, depending on when banks decided not to honor them, officials said.

In what appeared to be a possible break in the impasse, Minority Leader Robert C. Byrd (D-W.Va.) offered to complete Senate action on the budget plan by Thursday in exchange for a brief extension of the debt limit. Majority Leader Robert J. Dole (R-Kan.) met privately with Byrd and agreed to submit the offer to Senate Republicans today.

The issue must also go before the House, where Democratic leaders yesterday indicated increasing misgivings, despite earlier praise. Majority Whip Thomas S. Foley (D-Wash.) said Democrats were not necessarily committed to passage of such legislation, and Majority Leader James C. Wright Jr. (D-Tex.) said it was "fraught with misunderstanding and mischief."

The administration threat to allow government checks to start bouncing today, or Wednesday at the latest, came in a letter to Dole from Darman, who is acting treasury secretary while James A. Baker III is in South Korea for a meeting on international financial issues.

The government has neared the current amount it is allowed to go into debt, $1.824 trillion, and will not be able to pay its bills, including defense contract payments and interest on government securities, unless the debt limit is increased.

The pending legislation would raise the ceiling to $2.078 trillion, presumably enough to last a year. A short-term extension, opposed by the Reagan administration as well as congressional Republicans, could be enacted at any point, with long-term legislation to follow.

Yesterday Darman said, "As of this morning, we estimated that cash balances may be zero or negative tomorrow and will certainly be negative by Wednesday.

"When we formally determine that the next day's balance is to be negative, we will need to notify the Federal Reserve," he continued. "It is my understanding that, upon such notification, the Federal Reserve will then have to notify the banking system not to honor any government checks or electronic fund transfers . . . . Accordingly, all those with federal payment claims -- whether Social Security recipients or defense contractors or holders of government securities with interest payments due -- would then be unable to have those claims honored."

Darman concluded the letter with an oblique reference to the Federal Financing Bank, which sells securities to raise funds for some government agencies. It would provide the government what amounted to a "bridge loan," not subject to the debt ceiling but at a higher interest cost than other government financing, officials said.

The balanced-budget plan, advanced by Sens. Phil Gramm (R-Tex.), Warren B. Rudman (R-N.H.) and Ernest F. Hollings (D-S.C.), would set fixed deficit limits declining from $180 billion in fiscal 1986 to zero in fiscal 1991.

If the limits are not met, the president would be required to make across-the-board spending cuts to come within the ceiling.

Republicans argue that such drastic sanctions are needed to prod Congress to act. Democrats contend that the plan would give too much power to the president and threaten the economy in a recession.