Last Thursday night, nearly a year after promising huge deficit reductions and only a week after enacting legislation intended to force even larger ones, a groggy, sullen Congress was locked in bitter post-midnight combat over a modest part of a modest plan to achieve modest cuts.

"Who's in charge here? Why are we here?" cried House Minority Whip Trent Lott (R-Miss.) in an exasperated summary of the sentiments of lawmakers as they faltered even in their attempt to end the first session of the 99th Congress.

The scene Thursday night illustrated the yawning gap between Congress' obsession with reducing deficits and its ability to do anything about them. The issue at hand -- legislation that would have cut deficits by a modest $74 billion over the next three years -- was the year's clearest opportunity to actually do something about the red ink, but in the end this Congress couldn't do it. Its members left town without enacting that proposal, raising new doubts about their ability to deal effectively with any tough issue.

From start to finish, it was a Congress obsessed -- bedeviled by deficits that seem to hover around $200 billion a year no matter what was done to cut them and haunted by fear of how this debt could ultimately affect the economy.

There was also the growing, compelling fear that constituents, who now rate deficits as their most troubling national concern, are losing patience with antideficit talk that is not accompanied by antideficit action.

"We may have finally hit critical mass on Rotary Club speeches," said House Republican Policy Committee Chairman Dick Cheney (R-Wyo.), referring to waning public patience.

Congress had some substantial achievements as well as failures during the unusually long, 11-month session. The House passed sweeping tax-revision legislation, the full Congress enacted reforms of the nation's farm legislation, and the two chambers found compromises on difficult issues such as the MX missile and aid to Nicaraguan "contra" rebels. Pressure from Congress forced the administration to act on issues ranging from farm credit to trade and sanctions against South Africa.

But the session's real hallmark -- passage of Draconian legislation to force a balanced budget within five years -- was both an extraordinary achievement and a spectacular confession of impotence.

Having failed to cut spending enough to make much of a dent in deficits, and deterred from trying tax increases by President Reagan's adamant opposition, Congress threw up its hands and imposed a fiscal straitjack on itself and the White House.

According to one of the bill's chief sponsors, Sen. Warren B. Rudman (R-N.H.), the straitjacket was enacted only because all other deficit reduction efforts had failed. The most serious of them -- a proposal from Senate Republicans to impose new energy taxes and trim Social Security increases -- was shot down by the White House. "What it came down to is failure on all sides," Rudman said.

The final legislation, named for Rudman, Sen. Phil Gramm (R-Tex.) and Sen. Ernest F. Hollings (D-S.C.), sets targets for enormous deficit reductions each fiscal year between now and 1991, by which time the annual budget deficit is supposed to disappear. If Congress fails to meet those targets by passing spending cuts and/or new taxes, an automatic procedure will go into effect forcing reductions in all government spending programs except those given special protection in the law, mainly Social Security and poverty programs.

It is a "Look, Ma, no hands" approach to fiscal management, according to House Budget Committee Chairman William H. Gray III (D-Pa.).

Other lawmakers, including House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.), have suggested that the Gramm-Rudman-Hollings antideficit approach will collapse of its own weight as Congress encounters the political repurcussions of its enforcement. Still others, pointing to fizzled efforts at deficit reduction this year, said Congress and the White House have amply demonstrated their inability to act on the issue without outside constraints.

The most significant policy change forced by Congress this year was to stall Reagan's military buildup after a near-doubling of defense spending in the first four years of the administration. Defense still fared better than domestic programs, which were frozen or cut. But military spending hardly grew enough this year to cover the cost of inflation.

Defense faces even more troubles in the years ahead under Gramm-Rudman-Hollings. If the law works as intended, Reagan will have to take deep cuts in military spending to get the domestic cuts he wants, or swallow a tax increase.

"If he doesn't give on taxes or defense, there will be war from Day 1," said Rudman, who, like many other lawmakers, says he thinks that the president will have to give ground on both.

The power of the deficits' spell over Congress led not only to frustration, friction and paralysis in Congress but also to serious strains between Reagan and lawmakers of both parties, including Senate Republicans, who led the charge on deficit reduction.

Many members, including Republican leaders, complained that Reagan, preoccupied with other issues -- such as tax revision -- was indifferent to deficit reduction and failed to provide the leadership necessary to pull Congress together on the issue. Frequent mishandling of congressional relations by the White House, especially after Donald T. Regan took over as chief of staff, strengthened Congress' inclination to go it alone.

These were some of the other congressional actions this year:

*Both chambers passed legislation extending and expanding the "Superfund" for toxic-waste cleanup but stalled short of final action in a dispute over taxes to finance it and other issues. It was this dispute that made it impossible to enact the "reconciliation" bill requiring $74 billion in deficit reductions over the next three years. Extension of clean water and safe drinking water legislation also stalled.

*In a flurry of initiatives on trade, Congress passed but Reagan vetoed legislation restricting imports of textiles and other items. Congress delayed a vote on overriding the veto until August, in part to use the threat of an override to force a strengthening of the administration's stand in trade negotiations.

*The Senate approved a new version of immigration law revision; the House is expected to act on it next year.

*House committees approved legislation restoring major civil rights measures curtailed last year by the Supreme Court in the Grove City College case, but the measure has been stalled by a dispute over antiabortion language.

*The Senate approved an easing of federal gun controls, but the bill has been pigeonholed by the House.

*The House passed the largest water-projects bill in history, authorizing up to $20 billion in dams, waterways and related projects through 1998, and requiring local contributions to help finance them. A smaller Senate bill favored by the administration is pending in committee.