The Treasury Department yesterday borrowed $5 billion to avoid a default, using an unprecedented and complicated procedure to avoid exceeding the federal debt ceiling.

The Treasury said it was forced to use the procedure because Congress failed to pass legislation raising the ceiling on the amount it is allowed to borrow for use in paying its obligations.

The Senate, after hours of wrangling, could not agree to raise the amount the government is allowed to borrow from $1.824 trillion to the $2.078 trillion level the Reagan administration is seeking.

The Senate last night did pass a bill authorizing the Treasury to go ahead with the complicated procedure to borrow $5 billion from the Federal Financing Bank between now and Oct. 18. However, the House of Representatives rejected the FFB measure, saying that the Treasury already had the authority to use the procedure.

Treasury officials said unless Congress acted by midnight last night, it would go ahead with the procedure, previously described by high-ranking Treasury officials as "unprecedented and questionable." But congressional leaders made it clear the debt ceiling could not be raised before today.

"Only in the event that Congress fails to raise the current debt limit today will this procedure be used, in order to assure that the government can raise cash in order to avoid default," Treasury Secretary James A. Baker III said in a letter yesterday to Senate Majority Leader Robert J. Dole (R-Kan.).

Senate Finance Committee Chairman Bob Packwood (R-Ore.) said the Senate did not approve a short-term extension of the debt limit, which is usually done under these circumstances, because the Senate Democrats raised sharp objection to the bill. Until yesterday, the debt ceiling bill had been held up by debate over a measure that would force the government to balance the budget by 1991.

At the end of business on Tuesday, the government had virtually run out of money with $3 million cash left in its coffers, Baker said in his letter yesterday.

At the close of business yesterday, the Treasury's cash balance was estimated to have become negative -- absent the borrowing -- and the government's checks would have started to bounce.

What the Treasury did yesterday was to auction $5 billion in short-term bills, which, without the special procedure, would have put the U.S. debt over the ceiling.

After auctioning the debt, the Treasury said it would exchange some nonmarketable securities that it already held for securities of the Federal Financing Bank. The FFB securities are not subject to the congressionally-mandated debt limit, so using them to replace other Treasury debt instruments -- which are subject to the limit -- reduced the amount of debt the government holds.

With the amount of total debt reduced below the ceiling, the government was able to complete the auction of the $5 billion in short-term bills, Treasury officials explained.

Baker said in his letter to Dole that "we are reluctant to use the Federal Financing Bank authority in the manner that will be required -- in order to avoid default -- if the Congress does not raise the debt ceiling today. We appreciate that some members of Congress are similarly reluctant to see this FFB authority used."

The FFB was set up by Congress in 1973 to coordinate sales of government securities issued from independent federal agencies. It also has the authority to sell up to $15 billion worth of securities to the public.