Democrats made gains in the race for the House of Representatives last night, but it was not clear they had picked up enough seats to take back effective control of the House from President Reagan.

The Democrats were running well in districts where the unemployment rate was higher than the national average of 10.1 percent and in some parts of the South and West they were also gaining ground. But there were enough Republican victories throughout the country to cloud the picture early this morning.

Based on returns available at 2 a.m., it appeared that the Democrats would pick up about 20 seats. That could affect the shape of legislation in the next Congress -- many of Reagan's major victories in the last two years have been by fewer than 20 votes. But the president may still be able on many issues to forge the kind of conservative coalition that has won for him in the past.

Republicans lost seats in Pennsylvania, New Jersey, Delaware, Massachusetts, Virginia, New York, North Carolina, Alabama, West Virginia, and Ohio, but they also picked up seats in Indiana, Pennsylvania, New York, Connecticut, Florida and Nevada.

In Illinois, House minority leader Robert H. Michel claimed victory in his seesaw reelection race against Democratic union lawyer G. Douglas Stephens, in a district hurt both by unemployment and sagging farm prices.

Early today, Republican campaign officials listed 12 incumbents who had lost. They included Thomas B. Evans of Delaware, who was tarnished by his association with Playboy model and sometime lobbyist Paula Parkinson; Eugene V. Atkinson of Pennsylvania, the former Democrat who switched parties at the peak of Reagan's success last year; brassy freshman John LeBoutillier of New York; and Jim Coyne of Pennsylvania, who was vigorously opposed by groups advocating a nuclear freeze after he switched sides on that issue on the House floor. Coyne lost a rematch against former Democratic representative Peter H. Kostmayer.

Based on late returns, it appeared that several other incumbent Republicans had fallen, including freshman Rep. Ed Weber of Ohio and William C. Wampler of Virginia. In south-central Illinois, veteran Rep. Paul Findley (R) was narrowly behind lawyer Richard J. Durbin (D) with most of the vote counted.

But the pattern of the night was confusing, as the fate of Indiana Republican John Hiler showed. Hiler was declared a loser in his race against Richard C. Bodine early in the evening, but came back to defeat Bodine when final votes were counted.

In several districts where two incumbents were pitted against one another because of redistricting, Republicans were holding their own.

In Massachusetts, Democrat Barney Frank defeated Republican Margaret M. Heckler. But in New York, Republican Guy Molinari was the projected winner over Democrat Leo C. Zeferetti and Republican Benjamin A. Gilman was projected as the winner over Democrat Peter A. Peyser. In South Dakota, Democrat Thomas A. Daschle was projected as the winner over Republican Clint Roberts. In Missouri, Democrat Ike Skelton led Republican Wendell Bailey.

In Tennessee, Cissy Baker, daughter of Senate Majority Leader Howard H. Baker Jr., was a distant loser to Democratic public service commissioner Jim Cooper.

In Oklahoma, once-threatened Democratic Rep. James R. Jones, chairman of the House Budget Committee, was reelected.

But results from New Jersey were an example of the confusion that persisted as returns came in last night. There, the Democrats picked up the seat held by Republican Rep. Harold C. Hollenbeck. He was beaten by Robert G. Torricelli, who was an aide to former vice president Walter F. Mondale.

But in two other New Jersey districts the Democrats had hopes for, Republicans were doing well. Freshman Rep. Christopher H. Smith defeated Democratic state Sen. Joseph P. Merlino, while Republican Rep. Matthew J. Rinaldo was ahead of Adam K. Levin.

The Democrats were running very strongly in some of the high unemployment areas of the Midwest.

In Michigan, former Democratic representative Bob Carr was leading Republican Rep. Jim Dunn in a 1980 rematch in a district that includes Pontiac, where unemployment has been running at about 28 percent.

In Ohio, Republican Rep. Ed Weber was losing to former Carter administration aide Marcy Kaptur in Toledo.

In Florida, however, embattled freshman Republican Bill McCollum was defeating state Rep. Dick Batchelor. At the same time, Democratic Rep. Dante B. Fascell, a 14-term veteran of the House, had a commanding lead over television anchorman Glenn Rinker. In one of that state's newly drawn districts, Republican state Sen. Tom Lewis was leading lawyer-rancher Brad Culverhouse.

In Mississippi, Democratic state Rep. Robert G. Clark, seeking to become the first black congressman from his state since Reconstruction, was losing to former circuit court judge Webb Franklin.

Back on the Democratic side, however, returns from Texas in several contested districts showed Democrats running better than expected. In El Paso, Democratic state Rep. Ronald Coleman had a commanding lead over his Republican opponent, Pat B. Haggerty. In San Antonio, Democratic Rep. Abraham Kazen Jr. led Bexar County commissioner Jeff Wentworth. In a new district between Dallas and Fort Worth, Democrat Tom Vandergriff, former mayor of Arlington, Tex., was leading former Fort Worth city councilman Jim Bradshaw.

In a race that both parties had tagged as a weathervane in Connecticut, Democratic state senator William E. Curry Jr. apparently lost to state Sen. Nancy L. Johnson (R) for the seat of Rep. Toby Moffett, who was running for senator this year.

In two districts to which both parties were looking in Indiana, incumbent Democrat Philip R. Sharp came back after trailing in early returns and was leading Ralph W. Van Natta, former mayor of Shelbyville. Also, Republican Joel Deckard was having trouble with Bloomington Mayor Francis X. McCloskey.

But in another test district, Ohio's 12th district around Columbus, incumbent Democrat Bob Shamansky, a freshman, was trailing State Sen. John R. Kasich (R).

In the 17th district around Youngstown, an area of high unemployment, incumbent Republican Lyle Williams maintained a narrow lead over former state representative George D. Tablack, with half of the precincts reporting.

In Kansas, the second district seat apparently changed from Republican to Democratic as ABC projected former state house speaker Jim Slattery, a Democrat, the winner over former state GOP chairman Morris Kay in a contest for the seat of retiring Rep. Jim Jeffries.

There were no House elections yesterday in Louisiana or in two districts in Georgia. Under Louisiana's system, all eight incumbents were reelected earlier, while the elections in two Georgia districts were delayed until Nov. 30 because of redistricting litigation.

Going into yesterday's voting, Democrats controlled 241 seats in the House, Republicans 192. There were two vacancies.

Because of retirements, there were 210 Democratic and 167 Republican incumbents seeking reelection. In six districts, two incumbents were thrown together because of redistricting.

In addition, there were 58 races without an incumbent running, including 21 newly drawn districts created through redistricting. These were mostly in the South and West. They were among the most fought-over districts in the country.

The House races were regarded by both parties as especially important this year. Taken together they represented a kind of rough national referendum on Reagan's economic policies. They also, of course, were the key to whether the president will be as successful in Congress the next two years as he was the last.

Though the Democrats were in nominal control of the House, Reagan pushed his economic program through the 97th Congress fairly easily, partly because of strict party discipline among Republicans, partly because of defections to his side by conservative Democrats, mostly from the South and West.

These Boll Weevil defectors, a group of about 40 House members, helped deliver victories to Reagan on his first and second budgets and on his three-year tax cut, passed in 1981.

Few of the Boll Weevils were in competitive races last night, and despite occasional votes against the president, most of these conservative Democrats are likely to stick with Reagan on key economic votes in the next Congress.

The more crucial part of the coalition are the northern Republicans, or Gypsy Moths. What was impressive about Reagan's victories, especially in 1981, was not his support from southern Democrats but that fact that he lost almost no members of his own party. Many northern Republicans are far less conservative than the president, but still supported the main elements of his economic program.

Even before the election, there were signs of strain in the alliance, as some of the Gypsy Moths, who number about 30, bolted from the Republican leadership to help override Reagan's veto of a supplemental appropriations bill in September and opposed the White House on other votes.

Some Republicans abandoned the White House because they were in difficult reelection contests and might be expected to vote with the president again next year once they are safely reelected.

But the tenor of many campaigns this fall indicated there will be more independence among Republicans next year, especially on votes affecting defense spending and cuts in social programs.

Whatever the final results, their effect on the balance of power in the House is bound to be disputed, but the defeat of many GOP incumbents would prove costly to the White House because in most cases those Republicans will be replaced by more liberal Democrats who have run as opponents of Reaganomics.

White House sources were saying this week that it would take a loss of 19 GOP incumbents to endanger their House coalition.

The implication was that with anything less than a 19-seat Democratic gain Reagan could continue to work his will in the next Congress, when he will again be pressing for domestic spending cuts and an increase in the defense budget.

The Democratic leadership, however, was positioning itself on the other end of the interpretive spectrum. "The margin by which President Reagan won approval of his budget Gramm-Latta reconciliation package last year: 217 to 211 on June 26 1981," wrote Christopher Matthews, assistant to House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.), in a letter to reporters last week. "A switch by just four members would have been decisive." This year, the Democratic budget resolution was defeated 202 to 225, Matthews said, adding that "In other words, if the Democrats pick up the 11 or 12 seats we can expect, we should be in good shape to ensure some needed balance on fiscal/economic matters."