Charles Willoughby and his wife Jacquie bought their town house at Circle Woods, a development between Fairfax City and Vienna, four years ago.

Last year, they were startled to discover that a corner of the complex had been designated for public housing since 1976. They were further dismayed to learn that county authorities wanted the prospective low-income families to use the development's swimming pool and tennis courts.

The Willoughbys and other Circle Woods residents, who say they were ignorant of the plans when they bought their town houses, are fighting the $1.2 million project. The Circle Woods dispute is the latest in a series of neighborhood battles against public housing in Fairfax County, which has about 600,000 residents and 569 public housing units.

The battle is also shaping up as one of the first issues of the 1983 county elections. Republicans are hoping to make public housing a campaign issue, and in particular, hope to target Democratic Supervisor James M. Scott, who represents the Circle Woods area and is public housing's foremost supporter on the Fairfax County Board.

Scott received a first-hand view of the issue's volatility when he met recently with Circle Woods residents. Homeowners fumed at him, demanding that he support them or risk losing his Providence district seat. Scott steadfastly refused, angrily rising to his feet at one point and announcing: "I do not take well to threats."

The "real problem," he said later, is that "nobody wants it next door to them."

One Circle Woods resident who agrees with Scott, adds: "They Circle Woods homeowners think that they county housing authorities are going to bring in a ghetto like in Chicago or New York. Clearly if you look at the income requirements you can see that is not the case."

Beverly Magida, a leader of the protesting residents, sees the issue differently. "This is not so much a stand on public housing, as on fair government," she said. Magida said the group is not opposed to public housing per se. "We're an educated and enlightened group of people." But homeowners object to this particular project because it is "ill conceived," she said, and because residents were kept in the dark for so long.

Fairfax County Redevelopment and Housing Authority officials said the homeowners should have been informed earlier, but that it was the developer's responsibility to tell them before he sold the townhouses.

The developer has replied that until last year he was under the impression that he was not obligated to include the project on his property.

The homeowners, who paid about $70,000 to $95,000 for the townhouses, blame both the county and the developer, and their anger has at times been highly visible. The daughter of the developer had garbage dumped on her front steps for a week, forcing her to move from the complex.

According to the housing authority, the public housing will consist of 21 three-bedroom town houses, designed to mix with the existing town houses and clustered on one acre in the northeastern corner of the complex. No family with an annual income of less than $10,840 could live there. The maximum income would be $17,500 for a family of three and $21,900 for a family of six.

Housing authorities expect all the units to be filled by Fairfax residents, although they added that federal law prevents them from denying units to families simply because they are from another area.

The Circle Woods project would be the 13th public housing site in the county and would bring to 590 the total number of units the county operates. It has been on the books since 1976, when it was included as part of a rezoning request by the original Circle Woods developers.

The current developer, John F. DeLuca, bought the property in 1978 and said he was told that the public housing stipulation only applied if the development contained 292 or more units. Because DeLuca planned for 274 units, he said, he made no provisions for the public housing.

The housing authority subsequently informed him that he had to include public housing on the site. "I didn't believe it," DeLuca said. "If the IRS wrote and said you owed $100,000, would you believe it? Well, neither did I."

After the original sellers' attorney backed the housing authority's position in 1978, DeLuca changed his position and agreed to give the authority land at the rear of his development. He also agreed to try to persuade the homeowners association to allow the public housing tenants to use the pool and tennis courts. The housing authority said it would compensate the association for the use of the facilities.

The association is scheduled to vote on the pool and tennis issue next month. Homeowners complained that DeLuca holds a majority share of the votes, but he has said he would abstain if the residents are overwhelmingly against the proposal.

Those who attended the meeting with Scott made clear their opposition. "When an outsider crashes a party, uninvited, you don't invite them to join the family," homeowner Rick Norment declared, adding that the recreational facilities are too small to handle the additional people.

Circle Woods homeowners were equally angry at the housing authority, which, they asserted, had a responsibility to inform them of the proposal. The authority disagreed. "We can't sit in the sales office everyday," said spokesman Deirdre Coyne.

Residents also argued that tax money could be better spent by rehabilitating existing buildings for low-income housing, rather than by building new homes. Mary Egan, deputy director of the housing authority, said the Circle Woods money was federal funds set aside for new construction. It cannot be rechanneled into acquisition and rehabilitation projects, she said.

The housing authority has agreed to hold public hearings Jan. 11, to consider homeowners arguments against the project. Scott rejected the Circle Woods residents' request for a Board of Supervisors hearing. But Board Chairman John F. Herrity, a Republican, has said he plans to intercede for the group and request one in January.