Three incumbent congressmen were apparently defeated and another was in jeopardy in returns from yesterday's Illinois primary.

In the three-way fight for the Republican nomination for lieutenant governor, Gov. James R. Thompson's choice looked like the probable winner over a prominent women's rights advocate and an early supporter of President Reagan.

But Rep. Tom Railsback, a GOP moderate, was evidently defeated by state Sen. Kenneth G. McMillan, a staunch conservative, who charged Railsback with straying too often from the Reagan line.

McMillan piled up a large enough lead over Railsback in the rural counties that late returns from Railsback's own industrial Rock Island County apparently failed to save the eight-term incumbent. Railsback said early today that he believed he would lose by 2,500 to 3,000 votes.

In a contest devoid of that sharp ideological split, Rep. George M. O'Brien (R) defeated Rep. Edward J. Derwinski (R) in a race where redistricting pitted two old friends against each other. The new boundaries favored O'Brien and he won by piling up so big a margin in Will County that Derwinski could not overcome it in the territory left from his Cook County base.

In Chicago, two Democratic incumbents who had lost party endorsement for new terms were battling for survival. Rep. John G. Fary lost to Alderman William O. Lipinski, his younger challenger. But confusion and delays in vote counting clouded the outcome of Rep. Gus Savage's race with former Chicago Transit Authority Chairman Eugene Barnes and two others.

Voting was extremely light across the state, with estimates that barely a quarter of the voters turned out.

In the GOP lieutenant governor's contest, state House Speaker George Ryan, Thompson's choice, was running ahead of state Rep. Susan Catania, a feminist and party maverick. State Sen. Donald Totten, who sought to capitalize on his links to Reagan, was third. Totten conceded Ryan's victory but Catania clung to the hope that uncounted ballots in Chicago might give her an upset.

Chilly, damp weather in much of the state and the absence of any serious challenge to Thompson or his prospective Democratic opponent, former senator Adlai E. Stevenson III, made for a low turnout in the first primary of 1982.

Redistricting threw Derwinski, a senior member of both the Foreign Affairs and Post Office and Civil Service Committees, into the same district as O'Brien, a 10-year veteran of the Appropriations Committee.

Railsback, who played a crucial role in the Judiciary Committee impeachment proceedings against President Nixon, was locked in a classic moderate-conservative battle against McMillan. One of the leading GOP advocates of civil rights and legal services for the poor and restrictions on corporate campaign funds, Railsback faced his first primary opposition in 16 years.

McMillan, 39, had a strong base in rural parts of the district because of his legislative service and his previous employment as an aide to former secretary of agriculture Earl Butz and the Illinois Farm Bureau. He drew support from anti-abortion groups and national conservative organizations critical of Railsback's independence.

On Chicago's South Side, Fary and Savage were caught up in organizational battles. Fary, who was virtually appointed to Congress in 1975 by then-Mayor Richard J. Daley, found himself challenged by Lipinski, an ally of the mayor's son, Cook County State's Attorney Richard M. Daley (D).

Savage drew three challengers, including Barnes, who is backed by Chicago Mayor Jane Byrne (D), in a contest where the chief issue was Savage's 50 percent roll-call participation record in 1981--the lowest in the Illinois delegation.

But the main interest was in the normally invisible lieutenant governor's contest on the GOP side. Thompson put his prestige behind Ryan, a conservative from Kankakee who has carried the governor's program all six years Thompson has been in office. Ryan was challenged from the right by Totten of suburban Chicago, who organized the Reagan campaigns in Illinois in both 1976 and 1980.

Despite Thompson's effort to keep the White House neutral, Totten was aided in the campaign by Transportation Secretary Drew Lewis, Sen. Paul Laxalt (R-Nev.), Rep. Jack Kemp (R-N.Y.) and longtime Reagan adviser Lyn Nofziger.

But a greater threat to Ryan, according to pre-primary polls, came from Catania, a feminist liberal who backed John B. Anderson for the 1980 GOP presidential nomination.

As the former head of the Illinois commission on the status of women and the only avowed supporter of the Equal Rights Amendment in the race, Catania drew financial help from feminists across the country, who saw in her candidacy a last-ditch chance to push Illinois into the list of states ratifying the ERA.

Prominent opponents of ERA, including Phyllis Schlafly, endorsed Ryan over Totten--despite the fact that Thompson himself is an ERA supporter and Totten has been a vocal critic of the governor on that and other issues.

The Republican lieutenant governor's contest was the only major statewide contest in either party. Thompson, seeking an unprecedented third term, had only nominal opposition for renomination from V. A. Kelley, a mule skinner, and Dr. John E. Roche, a crusader for lower taxes. He defeated them easily.

On the Democratic side, there was no opposition to the slated candidates, Stevenson for governor, and Grace Mary Stern, a three-term county clerk in normally Republican Lake County, north of Chicago, for lieutenant governor. Early polls have given Thompson a narrow lead, but the battle of two of the state's best-known political figures, neither of whom has ever lost an election, is expected to be a close contest.

The squeeze in House contests resulted from the state's lagging population growth, which reduced the size of the Illinois delegation from 24 to 22 seats. As in many other states, a legislative impasse on redistricting forced the issue into the federal courts, where two of the three judges picked the Democratic plan. This saved all the Chicago Democratic seats and pitted two pairs of suburban Republican incumbents against each other.

One of the contests was resolved when 20-year-veteran Rep. Robert McClory, the ranking Republican on the Judiciary Committee, agreed not to seek reelection at 74 and yielded to 46-year-old John Edward Porter, who came to Congress in a special election in 1980.

Redistricting also played a key role in setting up the Railsback-McMillan primary. The territory the 16-year House veteran picked up included the state senate district of McMillan.