Despite his landslide reelection victory, President Reagan lost his tight grip on Congress over the last six months as resurgent House Democrats and rebellious Senate Republicans pulled in opposite directions to seize the legislative initiative.

As Congress started an August recess late last week, the Democratic majority in the House, initially plagued with uncertainty over the president's 49-state victory, was basking in the glory of its first real budget victory since Reagan took office in 1981.

The Republican-controlled Senate, which was first to champion the cause of deficit reduction, found itself seething over what it regarded as an unholy alliance between the White House and the House to torpedo tax increases and Social Security constraints that it had proposed as bold solutions to the problem.

All of that leaves Reagan facing a volatile political landscape -- an emboldened House and a resentful Senate -- when Congress returns after Labor Day to deal with what may be the most problem-packed agenda of his five years in office.

"For the rest of the session, the president is going to have a tough time," said House Budget Committee Chairman William H. Gray III (D-Pa.), who has emerged as one of the bright new stars of Congress in his commanding role in the budget deliberations.

Republicans agreed, warning that the problems are getting tougher as the White House, distracted by internal staff changes, the president's cancer surgery and other pressures, fumbled early opportunities to capitalize on Reagan's reelection and ruffled feathers among powerful leaders.

"It's going to be a tough two-and-a-half months," said Senate Majority Leader Robert J. Dole (R-Kan.).

In a fairly fast-paced start, this 99th Congress already has passed a reduction program for the fiscal 1986 budget and deficit, a wrap-up spending bill for this year and the first foreign aid bill since 1981. It has settled the controverial MX missile and Nicaragua aid issues, at least for the time being. It has nearly completed action on the defense bill for next year and is likely to give final approval to sanctions against South Africa when it returns in September.

Action is also under way on the farm program, most spending bills for next year, clean water reauthorization, the Superfund for toxic waste cleanup and a host of other major measures.

But the stakes will be higher and the pace more frantic in the coming fall session. Congress will tackle major tax revision, additional appropriations bills and a huge catchall spending bill to start the fiscal year Oct. 1.

In addition, it must act on the enforcement of program cuts imposed by the budget and must approve legislation raising the federal debt ceiling above $2 trillion. The fight over that move is certain to remind the country that the nation's debt has doubled under Reagan.

The volume of business and short time for handling it is a special problem for Reagan, who has done best when he can dominate the legislative agenda with a single powerful issue, such as his budget and tax cuts in 1981.

Buoyed by strong personal popularity in Congress as well as the country as a whole, Reagan continued to set the broad outlines of a largely conservative congressional agenda this year.

"He sets the broad outlines but he only seems to be effective at that," said Rep. Vic Fazio (D-Calif.), who was deeply involved in the budget fights of the last few months.

But on issue after issue on both foreign and domestic fronts, from his military rearmament effort to his squeeze on social welfare programs, he was forced, more than before, to settle for less than he wanted. And in the case of sanctions against South Africa, Congress moved simply to repudiate the administration's policy of "constructive engagement," or quiet diplomacy, with the Pretoria government.

The administration "had to take less than they wanted in most cases . . . . The president personally still has a lot of clout, but that doesn't always mean his legislative program is going to make it," Dole said in an interview after Congress recessed.

The MX missile was saved, but the program was cut in half. Aid to antigovernment rebels in Nicaragua was renewed, but not for military uses as Reagan wanted, and the Central Intelligence Agency was barred from direct involvement. Domestic programs were cut but not as much as the president wanted; those he wanted to eliminate were kept at least on life supports.

Defense spending probably provides the best example of how Congress is moving in Reagan's direction but at a far slower pace. Military spending will continue to grow next year, but only enough to cover the costs of inflation -- in effect leveling off his buildup. Reagan had sought to increase defense spending by 6 percent above inflation.

Reagan was able to use his veto power or the threat of it, combined with his skills at "Congress-bashing," to keep Congress from stepping over the lines he had drawn.

The point was made early on, when Congress approved emergency farm legislation over Reagan's objection. He quickly vetoed it and the House abandoned the bill.

His veto pen, and the prospect of facing White House denunciations, also eliminated all thoughts by Democrats of trying to expand domestic programs or embark on new initiatives.

In addition, Democrats appeared to be taking a harder line on national security and foreign policy, such as aiding anticommunist insurgencies, partly in response to the perceived popularity of Reagan's tough approach.

But Reagan could not get his way on some of his most controversial initiatives, even in the Republican Senate. Senate leaders abandoned efforts to expand the president's veto authority through a line-item veto after failing several times to shut off a Republican-led filibuster against it.

On critical issues such as taxes and Social Security, Reagan succeeded in exploiting congressional divisions to thwart revolts. But this often had the effect of helping House Democrats more than Senate Republicans.

In the opening days of the 99th Congress, congressional divisions fell along partisan lines, especially in the House, where the dispute over Indiana's 8th Congressional District seat bitterly split Republicans and Democrats for months.

The party-line vote to give Democrat Frank McCloskey the seat angered and unified the House Republicans, especially the large block of freshmen, and helped Reagan win on several key issues, including aid to Nicaraguan rebels and funding for chemical weapons.

But as the budget came to dominate the congressional debate, institutional divisions took over, with Democrats and Republicans in the House joining forces against Senate leaders' insistence that Social Security be included in any deficit-reduction package. Similarly, the Senate found few followers in the House for its proposal to raise taxes.

Reagan, who in his presidential reelection campaign had promised not to tamper with Social Security, ended up siding with the House, isolating the Senate and leaving its Republican majority feeling angry and betrayed.

The experience was especially painful for Reagan loyalists in the Senate, who had often taken politically risky positions to back up their president and felt "sold down the river," as one of them put it. Dole has suggested that it will be harder to keep the Senate in line behind Reagan in the future, especially on economic issues.

"Whenever you get on economic issues where you have disagreements, there's going to be trouble," he said.

The budget debate and divisions had implications as well for the 1986 congressional elections and the 1988 presidential race.

With 22 Republican-held seats at stake next year in the Senate, which the GOP controls by only a four-seat margin, reelection politics are going to be increasingly important in shaping legislation. For instance, the outcome in farm states alone could determine control of the Senate, a factor that hangs heavy over both parties as they agonize about the farm bill.

Republicans could be hurt by identifying too closely with Reagan, but they also are reluctant to stray too far from a president who remains extraordinarily popular among their constituents. Similarly, Senate Democrats remain fearful of being tagged as obstructionists or setting themselves up for a barrage from the White House. Minority Leader Robert C. Byrd (D-W.Va.), for instance, withheld his vote on the budget until he received multiple assurances from Dole that Reagan would not criticize the budget after it was approved.

In the House, Democratic leaders were elated not only at the final outcome of the budget but at the Republican infighting, which the Democrats hope to convert into GOP losses in 1986.

"They have squandered their mandate . . . . They won 49 states by saying everything was fine, just leave the economy alone. Now they can't even govern," said the chairman of the Democratic Congressional Campaign Committee, Rep. Tony Coelho (D-Calif.).

At the same time, the Democrats were upbeat about their own prospects, convinced they have shed an image prevalent just six months ago of a party in disarray and bickering over its future.

Democrats can no longer be seen as "committed to taxes, wasteful government and weak defense," said House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.).

Behind the scenes of some of these fights presidential campaigns for 1988 are developing, principally between Dole and Rep. Jack Kemp (R-N.Y.), who played a role in persuading Reagan to abandon Dole's plan for Social Security spending constraints. Kemp scored a point or two in the early skirmishing but at the expense of some sore feelings among his colleagues.

The budget fight also was a rite of passage for leaders of the two chambers: for Dole, who is just settling in, and for O'Neill, who is preparing to depart at the end of 1986.

In his leadership debut, Dole embarked on a high-risk strategy of prodding both Reagan and O'Neill to accept more in the way of politically painful deficit reductions than they wanted, principally through tax increases and Social Security cuts.

It failed in the end, although senators, including some Democrats as well as Republicans, appear to have rallied around Dole in a spirit of injured institutional dignity.

O'Neill, despite his announced retirement at the end of this Congress, remains firmly in control of the House, as the scramble for leadership posts proceeds around him.

At the beginning of the session O'Neill faced challenges to his power from a group of younger Democrats who, convinced that he was the wrong spokesman for the party as it set about improving its image, wanted him to play a less prominent role. By the end of the budget fight, O'Neill had reaffirmed his leadership and most of the strains with the younger Democrats had abated.