Illinois Gov. James R. Thompson has made a useful habit of staying out of internal fights inside his own Republican Party. In 1976, he ducked the Ford-Reagan battle on grounds that he was a candidate for election himself. In 1980, when he lacked that excuse, he still stayed neutral in the Illinois presidential primary.

But this year, as he prepares for his toughest race, a bid for an unprecedented third term against former U.S. senator Adlai E. Stevenson III (D), a nasty nomination fight for lieutenant governor has embroiled Thompson in a battle from which he probably cannot escape unscathed.

Ironically, the office itself is so inconsequential that the man who filled it for the first five years of Thompson's tenure, Dave O'Neal, quit last year, saying there wasn't enough to do to justify the salary.

The speaker of the Illinois House of Representatives, George H. Ryan Jr., then told Thompson he would like to be his running mate in 1982. Having already rejected Ryan's bid for appointment to a vacancy as secretary of state, and needing the speaker's help to maintain the shaky one-vote Republican majority in the politically crucial legislative session now under way in Springfied, Thompson found Ryan's overture an offer he could not refuse.

But the normally insignificant office turned out to have wide allure this year. Next into the race was state Sen. Donald L. Totten, a staunch suburban conservative who had never bothered to conceal his contempt for Thompson's progressive Republican tendencies. Totten had won a degree of renown as the original Reagan backer in Illinois. He ran Reagan's losing 1976 primary campaign and in 1980 shepherded him to victory in Illinois and two other Midwestern states.

A battle between Ryan, aided by the Thompson organization and his own legislative colleagues, and the Totten-Reagan network would have been divisive enough. But then a third candidate jumped into the contest and scrambled everyone's plans.

State Rep. Susan Catania is a white Republican who has represented a predominantly black and overwhelmingly Democratic legislative district on Chicago's South Side for the past 10 years. Head of the house committee on human resources and, until recently, the state commission on the status of women, she was also a John Anderson delegate to the 1980 Republican convention.

Catania is a maverick, one of a handful of House Republicans who voted against the GOP's redistricting plan (which, she says, would have cost her her own seat). A chemist and mother of seven daughters, she is the only one of the three candidates for lieutenant governor who is with Thompson in supporting the Equal Rights Amendment.

Catania says she is not a single-issue candidate, but tells audiences that her nomination would "send the strongest possible message to the governor and legislature about the importance of all the human issues, including ERA."

Because both Ryan and Totten oppose ERA and Ryan has been instrumental in blocking a rules change that would let it pass by simple majority vote rather than three-fifths, women's groups in and out of the state have rallied strongly to her support. The National Organization for Women (NOW) has loaned or contributed more than $65,000 to her effort.

Catania has an amateur campaign organization, run by her husband and brother-in-law, but she has raised over $200,000 for radio and TV ads, featuring actress Marlo Thomas.

Early-March polls showed most prospective Republican primary voters undecided, but Catania matched Ryan in name-recognition and support. Totten was running third, both in the polls and in fund-raising.

But last week Totten got a $130,000 campaign contribution from conservative businessman W. Clement Stone to launch a TV campaign of his own. And the possibility that Totten might split the conservative vote with Ryan, enabling Catania to win, was alarming enough to the Thompson organization that the governor gave $120,000 of his own campaign funds to Ryan.

The campaign has created problems for the Reagan White House, which values Totten as an old supporter but regards Thompson as a greater current asset in the negotiations over the president's federalism initiative.

Totten has been able to enlist Transportation Secretary Drew Lewis, Sen. Paul Laxalt (R-Nev.) and Rep. Jack Kemp (R-N.Y.) for fund-raising appearances in Illinois and is getting some (unpaid) counseling from Lyn Nofziger, another old Reaganite who recently left the president's staff to become a campaign consultant.

When Totten held a Washington fund-raiser last week, however, he said no one came from the White House. He blames Thompson for doing everything in his power to cut off aid from the Reaganites.

But anti-ERA activist Phyllis Schlafly, who is no Thompson ally, also endorsed Ryan, in order, she says, to stop Catania, and, according to Totten, to repay Ryan for his years of help in keeping Illinois from ratifying ERA.

All three candidates argue that they would strengthen the ticket as Thompson's running mate.

Thompson, who blames Catania for making this "a very emotional, polarized situation," contended in a recent interview that "most Republicans didn't want a contest for lieutenant governor."

But now that it exists, his aides say, Thompson has no choice but to go all-out to win it for Ryan, since a Ryan loss would undoubtedly be taken as a sign of Thompson weakness against Stevenson.

"I've endorsed him," the governor said. "I've campaigned for him. I've sent word to Republican organizations I want him. And I'm going to end up spending more money nominating him than I care to think about. But I am not going to lose."