President Reagan faces a less-compliant Congress than four years ago after Tuesday's election, in which modest conservative gains in the Democratic-controlled House were offset by small but potentially more far-reaching losses in the Republican-controlled Senate.

The results fell far short of the mark that Republicans aimed for when they went all out last week to duplicate their congressional sweep of 1980. That sweep led to stunning legislative victories for Reagan in his first two years in office.

In the House, Republicans picked up 15 seats, with another three still undetermined, thereby reducing the Democrats' margin to anywhere from 67 to 73 out of the House membership of 435.

That leaves the Republicans little more than halfway toward recapturing the 26 seats that they lost in 1982 when the Democrats took back effective control of the House from the conservative coalition that had dominated it and produced big victories for Reagan for two years.

Because some of the Republican winners replaced conservative Democrats who had often voted with Reagan, the ideological shift was relatively small, possibly only six to 10 seats. House Minority Leader Robert H. Michel (R-Ill.) pegged the conservative gain at about 10 seats; Democrats said they figured it was less.

In the Senate, Democrats made a net gain of two seats, reducing the Republican margin from 55-to-45 to 53-to-47. The tilt away from conservatives and administration loyalists may also have been stronger than the numbers suggest, some sources said.

In the most clear-cut defeats for the administration, Sen. Charles H. Percy (Ill.) and Sen. Roger W. Jepsen (Iowa) lost and were replaced by relatively liberal Democrats, Rep. Paul Simon (Ill.) and Rep. Tom Harkin (Iowa), respectively.

The election of Massachusetts Lt. Gov. John F. Kerry to succeed retiring Sen. Paul E. Tsongas (D-Mass.) kept that seat in liberal Democratic hands, as did the succession of West Virginia Gov. John D. (Jay) Rockefeller IV to the seat held by retiring Sen. Jennings Randolph (D-W.Va.).

Similarly, the election in Texas of Rep. Phil Gramm, a Democrat-turned-Republican, to succeed retiring Sen. John G. Tower (R-Tex.) kept that seat in conservative Republican hands.

Sen. Walter D. Huddleston (D-Ky.) was defeated by Republican Mitch McConnell in the only loss of a Democratic senator in Tuesday's elections. But that loss was offset by the election of Rep. Albert Gore Jr. (D-Tenn.) to fill the seat being vacated by Senate Majority Leader Howard H. Baker Jr. (R-Tenn.). Gore tends to tilt more to the left than Huddleston did.

"Moderate Democrats will remain in control of the House. And moderates, Democratic and Republican, will be in control of the Senate," said Rep. Tony Coelho (D-Calif.), chairman of the Democratic Congressional Campaign Committee.

"I don't think it's quite the kind of Congress that Reagan had in mind," added a House Republican aide.

People "should not expect too many victories" for the president, said a disappointed Michel, who said he feared the net gain in genuine votes was very small.

The elections not only failed to revive the bipartisan conservative coalition that took control of the House after Reagan's victory in 1980 but undermined Republican strength in the Senate, raising the possibility of a moderate-to-liberal bipartisan coalition on some key issues in that chamber.

Senate Minority Leader Robert C. Byrd (D-W.Va.) was quick to put out a bid for such an alliance. "The time is right for bipartisan cooperation," he said in an interview, explaining that he expects a winning coalition of Democrats and moderate Republicans on such issues as further social-welfare spending cuts.

The Senate was difficult for the Republican leadership to control before the elections, and there are other political forces now at work that could make it more unwieldy and fractious, one knowledgeable Republican suggested yesterday.

With the '84 elections casting some doubt about the reliability of Reagan's coattails and with lame-duck presidents normally having little clout in off-year elections, the 22 Republican senators whose seats will be at stake in the 1986 elections can be expected to be unusually independent, he said.

"The shots will be called by 22 people who will be saying to the president, 'Hey, I'd like to be loyal, but if it doesn't sell in my neck of the woods, forget it.' "

Moreover, the Republican said, a new majority leader and secondary-level party officials, who will be chosen in a caucus Nov. 28, will still be getting organized in the first few months of Reagan's second term, which many lawmakers believe will afford his only real opportunity this term for major legislative breakthroughs.

"For six months, he the new leader won't be able to hold them . . . and these may be the only six months that matter," said the Republican official.

"Ironically," he added, "the House could suddenly now be the place where the president turns to save his program."

In another upheaval in the ranks of the Senate Republican leadership, the defeat of Percy, who was chairman of the Senate Foreign Relations Committee, raises the specter of a ideological cleavage over the post, with spinoff effects on the majority leader's race.

The conservative Helms is next in line to succeed Percy, although he repeatedly assured North Carolina tobacco farmers during the campaign that he would not relinquish his chairmanship of the Agriculture Committee to take over foreign relations. He may not head both panels.

But he is under pressure from friends to take the Foreign Relations chairmanship, and some of his colleagues do not rule out a change of heart.

Morever, if Helms doesn't take the foreign relations post, it would go to Sen. Richard G. Lugar (R-Ind.). But Lugar wants to succeed Baker as majority leader and can't hold both jobs.

In line behind Lugar for the foreign relations chairmanship is Sen. Charles McC. Mathias (R-Md.), who is as much anathema to conservatives as Helms is to liberals.

There has been speculation that conservatives might vote against Lugar for majority leader to keep him as chairman of foreign relations and block the ascension of Mathias. A similar conservative maneuver blocked Mathias from becoming ranking Republican on the Judiciary Committee several years ago.

Other candidates in the race to succeed Baker, which is wide-open but expected to intensify quickly, are Finance Committee Chairman Robert J. Dole of Kansas, Energy Committee Chairman James A. McClure of Idaho, Budget Committee Chairman Pete V. Domenici of New Mexico and Republican Whip Ted Stevens of Alaska. Domenici, McClure and Stevens were up for reelection Tuesday and won easily.

In terms of voting balance in the Senate, a key question is whether Democrats, faced with gains that could help them regain Senate control in 1986 when Republicans will be defending nearly two-thirds of the seats at stake in that year's elections, will become more cohesive and assertive.

A more unified Democratic Party, picking up the half-dozen or more moderate Republican mavericks who frequently bolted from administration positions during the last four years, could become a much more powerful force on key issues in a second Reagan term.

One such issue is further production of the controversial MX missile, which hinges on another series of congressional votes next spring. With the modest conservative gains in the House, the MX conceivably could be approved in that chamber in light of the thin margins in votes on the missile this year. But conservative losses in the Senate, which also was divided sharply on the MX, raise doubts about whether the Senate would go along.

Another issue is the administration's beleaguered program for aid to anti-government guerrillas fighting the leftist Sandinista regime in Nicaragua. Such aid could gain support in the House while losing it in the Senate. But House leadership sources said they think there will be enough votes in the House to block a resumption of aid.