Caught between fear of defying a popular president and misgivings over his policies, a fractious 98th Congress went home to face voters Friday with a jumbled record of cooperation and confrontation that gives both President Reagan and the Democrats a measure of political vindication.

Especially as the session wound up in a spectacular Senate mud fight over debt-ceiling politics, attention focused on the partisan impasses, institutional deadlocks and political irresolution that hampered Congress in its relations with Reagan and further tarnished its image.

Less apparent were solid if modest achievements including a "down payment" on deficit reduction, a major expansion of wilderness areas, a comprehensive overhaul of federal criminal laws, enactment of women's equity programs, stronger cigarette labeling rules, a boost for generic drugs and legislation to encourage states to raise their legal drinking ages.

Almost forgotten as the session whimpered to a close was the bang with which it began, including enactment of a massive Social Security rescue plan and a major anti-recession jobs program.

Overall, however, it was a session marked by temporizing and fudging on a grand scale as lawmakers, frustrated in their efforts to get a grip on apparent swings in the country's mood, were torn between following Reagan and defying him.

In the end, the 98th Congress simply threw up its hands and bucked over to the 99th Congress a long list of tough issues ranging from immigration, civil rights and the "Superfund" for toxic-waste cleanup, to the MX missile, aid to antigovernment rebels in Nicaragua and a 35-year-old treaty outlawing genocide that has been ratified by 82 other countries and endorsed by Reagan.

In its relationship with Reagan, Congress was similarly irresolute, sometimes to the point of cutting a deal with him with one hand as it was poking him with the other hand.

From foreign policy and defense to tax increases and restoration of earlier social spending cuts, Congress over the past two years wheedled, cajoled and prodded a reluctant Reagan into accommodation -- even retreat -- on many issues.

But it rarely veered far off the general course laid out by the president at the start of his term, including a massive military buildup financed more by deficits than taxes and a squeeze on domestic programs that ended a period of government expansionism going back to the New Deal.

Instead, it created a situation in which effective control over the reins of policy in Washington shifted back and forth from the White House to Capitol Hill, sometimes in a bipartisan comity, sometimes in partisan rancor. Often lawmakers could be heard to complain that no one seemed to have a firm grip on the reins.

With the Democrats regaining effective control of the House at the midpoint in Reagan's term, coupled with a pratfall-style windup for the once sure-footed Republican leadership in the Senate, the 98th Congress at its end was nowhere near as compliant as the 97th Congress was at its start.

But it rarely moved to an open break with Reagan or the major thrust of his program. Often it fought its battles at the margin, split the differences or simply punted.

Its major achievements, such as the Social Security and jobs bills that were approved in early 1983, came only after a solid consensus developed that was rooted in undisputed political necessity.

Its major failures, such as the collapse of legislation to revise the country's immigration laws and its reluctance to go beyond marginal deficit reductions, reflected recognition of the political risks entailed in pitting conflicting constituencies against each other and moving before consensus developed.

In the broad spectrum between success and failure, often there was as much symbolism as substance from both the White House and Congress, especially as this year's elections approached and both sides tried to score political points rather than pass substantive legislation.

A deficit-reduction compromise of nearly $150 billion over the next three years finally was nailed down, although congressional budget projections continue to show deficits rising through the rest of the decade, even with the tax increases and spending cuts approved this year.

Final decisions on the MX missile and aid to anti-government "contra" rebels in Nicaragua were put off until next year, along with action on the Kissinger commission's ambitious long-term proposals for increased military and economic assistance to Central America.

Also put on the shelf were most difficult decisions on environmental policy, from new clean-water and clean-air rules to long-term financing of the Superfund, although waste-policy laws were strengthened and millions of acres were added to wilderness areas.

A farm program was patched together, including price-support freezes and a payment-in-kind program to reduce surpluses. But major revisions await another day.

New banking deregulation proposals were set aside, along with natural gas deregulation and offsetting efforts to curb cost increases. A compromise was struck over protectionism in a trade bill, but legislation to crack down on export of strategic goods died.

Congress and Reagan bowed to women on legislation to guarantee pension equity, curb child support abuses and examine pay comparability, but the proposed Equal Rights Amendment, once favored by both houses but opposed by Reagan, was put aside.

Without much encouragement from the administration, Congress extended and strengthened the Voting Rights Act but refused to reverse a Supreme Court decision that narrowed coverage of antidiscrimination laws, leaving that issue to be resolved next year, if ever.

Congress brushed aside most major "social-issue" causes of the New Right that got at least lip service from Reagan, including constitutional amendments to ban abortion and to permit school prayer. But it refused to ease restrictions on government-funded abortions and voted to give student religious groups "equal access" to public schools, with the House going one step further in taking a largely symbolic stand in favor of voluntary prayer in the classroom.

Neither house got around to approving Reagan's request for constitutional amendments to encourage balanced budgets and give the president a line-item veto for spending bills.

The House did some political muscle-flexing in passing legislation to require the president to submit a plan to achieve balanced budgets every year, but the Senate gave it only a passing glance.

With each passing year, Congress restored funds cut at the start of Reagan's term from social-welfare programs, especially those involving education, health and nutrition. And it acted this year to make it more difficult to remove people from Social Security disability rolls. But it has generally followed a policy of restraint in domestic spending that contrasts sharply with pre-Reagan years.

On defense, Congress can claim major reductions in Reagan's costly military buildup, including slashing his proposed after-inflation increase for this year from 13 percent to about 5 percent.

But Reagan can claim to have nearly doubled military spending since the year before he came to office, increasing the total to about $292 billion, a four-year hike of about $110 billion, including a boost of nearly $30 billion this year. Moreover, the heavy investment in major new weapons systems virtually assures continued big spending on defense in future years.

In foreign policy, Congress' impact was more indirect than direct, substantial but rarely decisive. Mounting congressional opposition to U.S. military involvement in Lebanon contributed to the administration's decision earlier this year to withdraw U.S. troops. Doubt remains among many lawmakers, however, over the effectiveness of Congress' invocation of war powers restrictions for the first time. Political pressures appear to have been stronger than institutional sanctions.

After months of trimming military aid requests for El Salvador and tying them up with strings to encourage human-rights compliance, Congress breathed a collective sigh of relief with the election of Jose Napoleon Duarte as president in May and approved nearly all the administration wanted for El Salvador this fiscal year.

But, still smarting from embarrassment over the CIA's involvement in mining Nicaraguan harbors, Congress rejected an administration request for additional aid to the contras last summer.

Then, last week, it agreed to delay further funding until at least March, when a majority vote of both houses would be required to release $14 million for the contras, half what Reagan wanted.

In what could be a metaphor for the 98th Congress and its relations with the White House, the Nicaraguan aid issue was left hanging, along with the MX question, in a compromise catchall spending bill that also put off until next year the issue of billions of dollars worth of domestic water-development projects.

On Nicaragua and the MX, the Democrats could claim victory. On water-project spending, Reagan could claim victory. But the net result will be a collective post-election headache for the president and the 99th Congress after it convenes.

Among the bills that died with the 98th Congress was one to release highway funds that include money from a gasoline tax increase approved two years ago to help the country recover from recession by pumping money into public works projects. The recession has long since given way to boom times in most of the nation, and much of the money has yet to be spent.