Charles Colson, the former aide to Richard Nixon and a convicted felon, had been kidding when he said the crowd gathered for the Arlington County Board meeting yesterday looked like it had "come for a lynching."

But after a wildly emotional three hour meeting, Colson said he may have been more accurate than he'd guessed.

To the cheers of standing room only crowd of about 120, the board voted 4 to 1 to reject Colson's request for a special zoning permit that would have enabled the nonprofit Prison Fellowship he heads to convert a church into its national headquarters.

"After listening to the crowd today," he said, "I think I'd be more concerned about bringing my inmates in contact with them."

Colson, the County Board and middle class residents of an Arlington neighborhood along Wilson Boulevard clashed yesterday over the issue that the various sides saw in different ways. The board tried to keep its attention on the zoning regulations that Colson's proposed purchase of the Calvary Church of the Nazarene might violate. Colson and other speakers pleaded for the special permit on moral issues involved in working with the imprisoned.

Colson, who served time for a Watergate related crime, was the most eloquent speaker of the afternoon. He began by apologizing for being nervous, saying, "The last time . . . I was in a courthouse I got one to three years."

At that point board member Walter H. Frankland Jr. shot back: "I must warn you, you're on tape." It was the last laugh of the afternoon.

The neighbors of the church saw the issue as one of protection from the worst elements in society. Despite Colson's assurances that the prisoners who would be served by his group would be born-again Christians and mostly white collar offenders, opponents conjured up visions of ex-inmates terrorizing their neighborhood of single family homes.

"A criminal is a criminal, whether educated or not. Let's face it," said Karl Davey, who said he has been a resident in the neighborhood since 1945. "They should be shown no leniency by giving them country club privileges."

Dennis Clague, another neighborhood resident, said the issue was children. "I want to give my kids the best environment I can. I don't think that involves the Prison FellowsIP."

Colson challenged the board, saying, "I've been in politics long enough to know how to count votes," "I hope this board can rise above the petty interest of political neighborhood views." The crowd hissed.

As the meeting wore on, tempers flared. The audience heckled the few speakers supporting Colson's proposal.

Board member John Purdy, the only member who voted for granting the use permit, castigated the audience at one point, saying, "I am very disappointed in what I've heard today." Immediately and in unison two dozen voices roared back, "So are we."

The fellowship now has offices in a McLean building but has a $1 million contract to purchase the Arlington church and two houses on the 3 1/2-acre church site.

The Rev. James Bearden, the current pastor of the church, said his congregation has purchased 23 acres of land on Little River Turnpike in Fairfax County for a new church. They have planned to remain in Arlington for two years sharing the facility with Colson's group.

With the permit denied, Bearden said he did not know who else might buy the church which has been appraised at $1,495,000. Bearden said in the 12 months the church had been for sale only Colson's group had made a legitimate offer. He worried aloud that finding another buyer will be a problem.

At that point someone in the audience snapped: "That's your problem."

Colson, considered by many to have been the Nixon administration's hatchet man, pleaded guilty in 1974 to a charge of obstructing justice by releasing derogatory information about Daniel Ellsberg, a principal in the Pentagon Papers case. Following his plea, Colson embraced a religious conversion.

He subsequently served seven months of the one to three year prison sentence he received in the case. Yesterday, Colson appeared confident as he stood before the County Board in a dark blue suit, occasionally raising his arms in a ministerial manner.

"Let's be big enough in Arlington County to start [a program] right here," he said. "Give people a chance, a little forgiveness and love, and you'll be surprised."

But the four board members who voted against Colson's proposal said his group's use of the church would be inappropriate. "The location is wrong," said Frankland. "It's against zoning and against the wishes of the community."

Colson said his Prison Fellowship, founded four years ago after his release from prison, had held 18 seminars for soon to be released prisoners in the Washington area, all without a single incident.