Rep. John B. Anderson, third-ranking member of the House Republican leadership and a leading figure in the liberl wing of the GOP, finds himself in an unexpectedly close fight for renomination in next Tuesday's Illinois primary, which opens the 1978 voting season.

Donald M. Lyon, a conservative minister, has attacked Anderson with a series of television ads alleging absenteeism and neglect of local concerns. Anderson backers here and in Rockford, the largest city in his 16th Congressional District, are worried that the chairman of the House Republican Conference and his organization are not responding adequately to the first serious challenge he has faced since his original House election in 1960.

Anderson has described himself as the test case for a "purge" effort by New Right organizations, which have contributed about $11,000 to the campaign of Lyon, a fundamentalist minister making his first foray into elective politics. Overall, Lyon has reported receipts of $140,000 and Anderson, $111,000.

But both Lyon and Anderson backers in Rockford said in telephone interviews that Anderson's problems result mainly from Lyon's skillful exploitation of the incumbent's political estrangement from his district, and Anderson's seeming inability to mount an effective organizational or personal response.

While Lyon has hit Anderson on such emotional issues as the Panama Canal treaties, abortion and gun control, Republican officials said the challenger's most effective weapons' are new television spots that began running last weekend.

One, showing a jet plane leaving Washington, accuses Anderson of having missed 264 votes in the past two years and says, "It's not just your foreign junkets we object to, John Anderson. It's the work that you're not getting done."

Another, on a similar theme, says, "Here in the 16th District, we believe in a day's work for a day's pay. You may have forgotten that, John Anderson . . ."

By contrast, Anderson's commercials have featured endorsements of the veteran congressman from district citizens and low-keyed comments by Anderson on his service.

But the incumbent - esteemed as one of the most effective orators in Congress and a spokesman for his party on energy, foreign policy and civil rights - has reacted angrily to some of Lyons charges in their joint appearances.

"Lyon has found John's vulnerable spot in attacking is integrity and he has made him explode, said one local Republican official who supports Anderson.

As word of the threat to Anderson's tenure spread in Washington, a number of leaders of the party's conservative wing have volunteered to help their colleague.

Rep. Jack Kemp (R-N.Y.), a conservative who often differs with Anderson on intraparty issues, flew out to Rockford Tuesday to commend him as "one of the most enterprise system" in Congress.

Earlier, former President Ford, former Secretary of State Henry A. Kissenger, and Republican National Chairman Bill Brock had spoken at fund-raisers for Anderson.

Lyon attempted to counter their influence in a mailing to Republican voters last week by conservative direct-mail specialist Richard Viguerie. The mailing referred to a "letter of endorsement" from Ronald Reagan, in which the former California governor expressed regret athis inability to appear at a Lyon fund-raiser and said, "I am with you all the way."

Lyn Nofziger, Reagan's chief political adviser, said the Reagan letter "was a personal letter. They never asked us to use it and we regret it that they did."

Anderson said yesterday that "I'm in good shape in Rockford [which casts about half the primary votes], but in the rural counties, he worked over the Panama Canal and gun registration issues very thoroughly and those TV ads have had some bite."

Some professionals who have visited the district find Anderson's telephone canvass and precinct operations suspect. But the incumbent's polls reportedly show him winning the Republican nomination with 55 or 60 percent of the vote.

One pro-Anderson Republican official said, "Lyon has run a 10 times better campaign, but I think John will pull it out. The Republicans here want John to win - by about 100 votes. They want him scared, but I think they know how he is scared."