Long-time residents of the exclusive Montgomery County enclave of Somerset, afraid that they may be out-voted when three, 20-story luxury condominium buildings are constructed on the only unbuilt lot in town, are considering de-annexing the 18-acre development area.

The Somerset residents, all of whom live in upscale single-family, detached homes, said they believe that de-annexation, a rarity for most tax-revenue-starved jurisdictions, may be a way to negate the impact of what they foresee as an influx of condominium residents with incompatible interests.

Somerset Mayor Walter J. Behr said that "de-annexation is something we've considered a possibility for as long as development of that parcel has been under discussion, some 25 years. But now the issue is being taken much more seriously.

"We're afraid the interests of the condo people will differ from ours, and, when the buildings are fully occupied, they'll be in a position to take over the town government," Behr said.

A 14-member committee is studying the de-annexation issue and expects to report to the Town Council by June, according to chairman Charles Edson. "The arguments against de-annexing are obvious: losing tax revenues. . . . The argument in favor of it is mainly fear of losing control. That's seen as a major concern, a major problem in many people's eyes."

The community of Somerset now is composed exclusively of single-family homes -- 410 of them. Only about 20 new homes have been built in the town in the past quarter century, and many date from the 1880s when the first 33 frame houses that made up the original 1906 incorporated town were built.

If the maximum approved total of 639 apartments are built, some of the 1,200 existing residents worry that the new residents will not want to pay taxes to maintain streets and parks, or a community pool and tennis courts that they may use infrequently, if at all. At the moment, 588 units are planned.

Moreover, the condominium dwellers would have the voting power to take over the town council and change the tax structure -- if they vote monolithically, Behr and Edson said.

The town -- bounded by Wisconsin Avenue on the east, River Road and North Park Avenue on the south, Little Falls Parkway on the west and Cumberland Avenue on the north -- has fought intense and often bitter legal battles during the past decade to retain its residential character and maintain its image as one of Montgomery County's most desirable addresses.

Prices on homes in Somerset currently range from $200,000 to $750,000, according to real estate broker and resident Donna Evers. The high price tags reflect a real estate market that puts a premium on close-in communities like Somerset that are semirural in character, but within walking distance of the Friendship Heights subway station at the District line.

The future of the so-called Bergdoll tract, where the condominiums are being built, is just off Wisconsin Avenue and south of Dorset Avenue, was central to this long-raging development debate, with the de-annexation study just the latest twist.

In a futile court fight to keep the Somerset House condominiums out of their community, residents argued that the county's zoning ordinance is illegal and that the high-rise buildings would dwarf their detached homes.

"We fought their specific development plans for years, but we lost, and we're now analyzing how to best live with the impending development," Behr said.

Plans call for construction of the Somerset House condominiums in two phases on land across from Saks Fifth Avenue's Wisconsin Avenue store and north of the Chevy Chase Building, according to county planners.

The first phase, with a 300-car underground garage and 159 luxury units, is under construction, planner Renee Kemp-Rotan said. The remaining two high-rise apartment towers -- one with 193 units and the other with 236 apartments -- will follow, depending on how quickly the first batch of luxury units sells, according to Behr.

A spokesman for the Somerset House Corp. declined to comment on the project, the de-annexation issue or when the first units would be available. He said the condominiums are being marketed as the "ultimate in residential luxury."

If the condominiums are unwelcome in Somerset, the adjacent village of Friendship Heights may offer them a friendlier haven.

"It's a very delicate and diplomatic situation, but we are interested in the Bergdoll parcel if Somerset should decide to de-annex," said Arthur Muller, chairman of the Friendship Heights council. "We do not want to poke our noses into Somerset's business, but we would like to discuss this possibility."

Friendship Heights is a special taxing district within Montgomery County that includes a triangular area immediately south of the Somerset House condominium project. "We are a high-rise community" consisting of about 5,000 residents in eight apartment towers -- all about 18 stories high -- two office buildings, a hotel and chic shopping strip along Wisconsin Avenue, Muller said.

Behr, who was elected Somerset mayor for the second time two weeks ago after a four-year hiatus, said he wants the de-annexing issue decided by the time his term is up in 1988.

James Yankauar, a member of the Somerset group studying the de-annexation option, said the committee has yet to develop "hard figures" on the potential tax revenue that the new condominium residents could generate for Somerset's coffers.

Under the town's current tax rate of 56 cents per $100 of assessed valuation, a $200,000 house generates $487 in municipal property taxes because taxes are levied on 43.5 percent of its full value, he said.

"Using this formula, it is easy to see luxury condominiums selling for $300,000 would generate considerably more," he said.

Municipal property taxes are the town's largest source of revenue, Behr said, and a major roads improvement project is planned over the next few years. The second-largest source of income is a rebate the town gets from Maryland on a portion of state income tax paid by Somerset residents.

If de-annexation still is a serious option once the committee's study is finished this summer, timing of the move is essential.

The town wants to retain as much control as possible throughout the construction of Somerset House and could conceivably postpone the move until all three buildings are occupied, reaping the tax revenue until sheer numbers show that the existing residents are in danger of being out-voted by apartment dwellers.

"The town wants the money, but doesn't want the condo owners to vote," said one resident of nearby Chevy Chase, who asked not to be identified.

The legal procedure for de-annexing is not well known -- annexing more property by far being the more popular move by municipalities. But Larry Speelman, county attorney for Frederick County, said the town of Middletown, Md., did exactly that a few years back by amending its town charter.

Middletown town clerk Beverly Dixon said developers wanted to build on a farm at the edge of town, "but there was a sewer moratorium in effect. The developers requested that we de-annex them so they could put in a separate well and septic system. So we did."

Speelman said a town's land area can be decreased by "reversing the annexing process and amending the charter the same way."

If Somerset decides to go the route of Middletown, the issue will come to a vote at a Town Council meeting only after several public meetings. Residents who disagree with the outcome can petition for a town referendum, Behr said.