When an Arlington church wanted to sell its property last year to a group of former prisoners organized by Watergate's Charles Colson, the neighborhood erupted in a protest and killed the deal.

This year the church's board of trustees negotiated a $1 million offer from what they had hoped would be a less controversial buyer: the Libyan government.

Now the leaders of the Calvary Church of the Nazarene are fearful that their carefully nurtured deal once again will be killed by a political storm across the Potomac. Troubled by the furor over Libya's relationship with President Carter's brother Billy, the church's pastor, the Rev. James Bearden, went before the congregation this week to deny any previous ties with either the Libyans or Billy Carter.

"This is such an emotional issue," said Bearden yesterday about Billy's Libyan connection. "This could be the thing that causes the congregation to vote it [the sale] down."

According to Bearden, the Libyan government has offered church fathers between $1 million and $1.5 million for their facilities, located on more than three acres of land along Wilson Boulevard.He said the Libyans plan to use the facilities as an elementary school for 200 Libyan children and did not rule out the possibility that the group might want to take possession of the property as soon as next month -- possibly leaving his congregation without a meeting place until it can build a new structure.

For Bearden, the struggle to get this offer from the Libyans has been long and intensely personal one. He and his congregation have been seeking to sell the church for more than two years so that they can build a new church on a 23-acre plot they have purchased in Fairfax County -- a spot that he says is closer to the homes of the members of his fundamentalist congregation.

He would hate to see his dream go down the drain -- as did the deal with Colson's nonprofit Prison Fellowship last year -- just because the prospective buyers were unpopular with the neighbors.

"If I go out and list my house for sale with a real estate agent, I would never question who they sell it to," he said. "It's for sale . . .

"What we've got here is a building that's for sale, which is only a building when it is no longer a church.

But as Bearden's 500-member congregation was humming with talk about the proposed sale, a spokesman for the Libyan government in Washington was professing ignorance of the offer.

Efforts to reach Richard Shadyac, the foreign agent for Libya who had reportedly arranged the deal, proved unsuccessful.

Bearden said the church has encountered many problems in finding a new owner because its million dollar price tag effectively placed it out of the reach of most groups that might seek an older building in an established sector of Arlington. "We heard from a few Oriental groups, but they just didn't have the money," he said.

Neighbors in the church's middle-class neighborhood were obviously not swayed by similar arguments from Bearden last year. Alarmed at the prospect that Colson's Prison Fellowship might bring in ex-inmates to the area, they succeded in quashing a zoning change that would have allowed the church to be converted into the fellowship's national headquarters. The area is currently zoned for residential use only, although churches are permited to operate as exceptions to that rule.

The proposed sale to the Libyans has been kept very quiet, which could perhaps account for the fact that neighborhood outcry against the idea has been kept to a minimum. But county officials caution that even if church fathers win the approval of the Libyan contract, it is possible the deal could again get hung up in the zoning process.

"If the primary use of the building is to be a school, they'll need a use permit -- which is approved by the county board," said Arlington zoning administrator Van Caffo, adding that neither the church nor the Libyans have asked the county whether the land could legally be used as a school.

"I would hope somebody comes in and talks to us about it first," he said. "So often people go and do things without finding out what they can do."