The 96th Congress adjourned itself into history last week without approving home rule extensions sought by D. C. Mayor Marion Barry, including a more generous federal payment formula and expanded autonomy for the city's judicial system.

Left to die were the long-sought financial reform provisions, and proposals to transfer full responsibility for the city's court system from the federal to the District government, and to transfer the ownership and management of St. Elizabeths Hospital -- the sprawling mental facility in Southeast Washington -- from federal to local control.

The adjournment of Congress along with the impending switch from the Carter to the Reagan administration leaves the city uncertain of its future relationships on Capitol Hill and at the White House.

Barry is scheduled to announce his 1981 legislative program soon. Two of its most important items are expected to be a renewed proposal to create a formula for establishing a rising federal payment and authorization for the city to borrow funds through the Federal Financing Bank to pay off much of the city's $409-million deficit.

Barbara Washington, the mayor's aide in charge of drafting and lobbying for legislation, said in an interview that she has not found anyone on President-elect Reonald Reagan's transition team who is dealing with the problems of the nation's capital. Barry, after a recent social contact with Reagan, said he expects a good relationship with the new administration.

Soon after becoming president in 1977, Jimmy Carter established a task force on District matters headed by Vice President Walter Mondale. The task force recommended that Congress enact several measures dealing with the capital's problems.

Heading that list was a proposed constitutional amendment to give residents of the city voting representation in both the House and Senate. That was approved by Congress, but has been stalled -- perhaps fatally -- in the quest for ratification by the necessary 38 state legislatures. To date, only nine states have approved the amendment.

The District is represented in Congress by a non-voting delegate, Walter E. Fauntroy (D), who -- despite a political rivalry with Mayor Barry -- has spearheaded most of the city's congressional initiatives. Most District legislation is overseen by a House committee, on which Fauntroy serves and has a vote, and by a Senate subcommittee.

Subcommittees of the House and Senate Appropriations committees oversee and enact the annual city budget and, in the process, set the level of the federal payment that is made in lieu of taxation on exempted federal and foreign embassy properties. The payment also is intended to defray unusual, federally related costs to the city, such as police overtime pay for political demonstrations directed against the federal government.

The city's 1975 Home Rule Charter, which authorized the popular election of the mayor and City Council, set a limit of $300 million for the federal payment. The 1981 budget enacted by Congress recently approached the maximum with a record payment of $296.4 million.

Fauntroy and Rep. Ronald V. Dellums (D-Calif.), the House District Committee chairman, introduced and held hearings this year on a proposed formula that would link the federal payment to the size of the city's tax collections each year. In 1982, the payment under their bill would be an estimated $376 million. However, the measure fell by the wayside for lack of legislative consideration, leaving it an open issue for next year.

Meanwhile, Rep. Stewart B. McKinney (R-Conn.), ranking minority member of the District Committee, has said he would reintroduce legislation next year to permit the District to levy a now-prohibited tax on the incomes of suburban residents who work in the city -- the so-called commuter tax.

In the past, such proposals have died, largely because of opposition from Rep. Herbert E. Harris (D-Va.), a District Committee member, and Rep. Joseph L. Fisher (D-Va.), a member of the tax-writing House Ways and Means Committee. Both were defeated for reelection last month, leaving the congressional fate of the tax in question.

Before the Nov. 4 general election, aides to Ronald Reagan said he would "probably" oppose a commuter tax, but there has been no word since. Reagan took no position on another key measure, judicial autonomy.

A bill introduced last summer at the city's request would transfer the right to appoint judges of the D.C. Superior Court and the D.C. Court of Appeals from the president to the mayor and switch the prosecution of all local criminal cases from the U.S. Attorney's office to a newly created D.C. Attorney General's office. It also would shift courthouse security from the U.S. Marshals Service to local control.

Many local judges have opposed the plan, which did not move forward after one day of hearings last summer. It is sure to be revived next year.

The transfer to St. Elizabeths to the city, a plan with roots in the Nixon administration, has run into problems because of the city's financial crisis, officials said. The proposal has been sidetracked, at least temporarily. Carter's proposed federal budget for the 1981 fiscal year had recommended that the hospital be put under the control of a joint federal-District public corporation to which both levels of government could contribute support.

Despite these setbacks, home-rule advocates did chalk up some victories. Congress enacted legislation providing federal support of pension programs for District teachers, police, firefighters and judges; a change in the Home Rule Charter that generally shortens the period of time that Congress has for reviewing legislation enacted by the City Council; an elimination of the presidential right to review mayoral vetoes of council actions; an extension of the city's right ot borrow money from the U.S. Treasury for its public-works program; the payoff of $20 million worth of bonds that were sold to build RFK Stadium; and a procedure for annual payments of the U.S. government's water and sewer bills to the city.