The Treasury Department, contending its hand has been forced by Congress' failure to raise the federal debt limit, has begun moves to dip into the Social Security trust fund and two other federal pension accounts to raise $17 billion to pay the government's bills.

Fearful that this would severely harm the Social Security trust fund, congressional leaders last night tried to engineer a one-week debt limit increase that would avoid the need for the Treasury action. But the effort collapsed after it was opposed by Senate Majority Leader Robert J. Dole (R-Kan.).

The complex bookkeeping transaction planned by Treasury was sharply criticized by congressional Democrats yesterday, who contended it could end up costing the trust funds $1 billion to $2 billion in lost interest payments over the next five years. In addition, a coalition of elderly and labor groups last night filed a lawsuit charging that the Treasury plan was an illegal raid on Social Security and seeking a temporary restraining order barring the Treasury from proceeding.

A federal judge, however, refused last night to grant the restraining order. But U.S. District Judge Thomas F. Hogan did not dismiss the lawsuit brought by the American Association of Retired Persons and others, saying it could be pursued if Congress does not resolve the debt ceiling problem today.

Treasury officials acknowledged their plan was an "extraordinary step" but insisted that without an immediate increase in the $1.8 trillion debt limit, the government will run out of money and 36 million Social Security beneficiaries as well as 3 million civil service and railroad retirees would be unable to cash their monthly checks.

"What's the alternative?" said Acting Assistant Treasury Secretary John J. Niehenke at a hearing before a House banking subcommittee. "The alternative is not paying the Social Security beneficiaries in early November."

Normally, the Social Security trust fund is held as a reserve and actual payments to beneficiaries payments are made from general Treasury revenue. The Treasury then reimburses itself from monthly Social Security payroll tax receipts.

But because the government is currently out of cash, the Treasury has decided to dip into the trust fund reserve. Niehenke said the Treasury's plan calls for it to cancel about $17 billion in Treasury securities held by the Social Security trust fund, the Railroad Retirement Fund, and the Civil Service Retirement Fund -- an action that would immediately place the government below the congressionally mandated debt limit of $1.8 trillion, allowing it borrow new cash from the public.

In anticipation of the move, the Treasury auctioned $17.75 billion in bonds and notes on Wednesday and plans to issue the certificates starting today, a Treasury spokesman said. But Treasury said its plan was only a stopgap measure that will keep the government afloat until Nov. 15.

Rep. Jim Jones (D-Okla.), chairman of the House Social Security subcommittee, said the fund could end up losing $300 million a year in interest because the Treasury will be reimbursing the fund with bonds at lower interest rates than those that it is cashing in today.

Jones also charged yesterday that Treasury, without any public announcements, "has already cashed in as much as $15 billion in long-term securities this fall, and that they may have done the same thing in 1984."

Jones called for a General Accounting Office probe of these earlier transactions. Congress needs to know "whether the Reagan administration has been secretly and illegally 'invading' the Social Security fund," agreed House Speaker Thomas P. O'Neill Jr. (D-Mass.).

Treasury officials denied they have illegally dipped into the trust fund already, but acknowledged that they redeemed $14 billion in long-term bonds in the fund to help meet Social Security payments in early October. The debt limit was first hit on Oct. 1.