The Alexandria City Council, in a surprise move, ousted a prominent black member of the city's school board yesterday and replaced him with a member of the politically conservative Alexandria Taxpayer's Alliance.

The council, voting early yesterday morning, removed the Rev. John O. Peterson, pastor of the Alfred Street Baptist Church and a four-year board member, amid charges that some school board members had secretly lobbied against the frequently outspoken minister.

The council's choice to replace Peterson, Norman Draper, a self-described conservative independent, leaves the nine-member school board with only two black representatives in a 10,794-student school system that is 48 percent black. William Euille, a black incumbent, was reappointed. The second black, board chairman Shirley Tyler, was not up for reappointment this year.

"There is going to be a tremendous repercussion in the black community," warned Councilman Nelson E. Greene, the council's only black member, after the vote.

Greene, who lobbied hevily for Peterson, appeared stunned as Mayor Charles Beatley read off the results, announcing that Euille and Judy Fever had won reappointment. Of the three school board members up for reappointment, only Peterson failed to win the necessary votes. All three Republican council members -- Vice Mayor Robert L. Calhoun, Margaret B. Inman and Carlyle C. Ring -- favored other candidates, denying Peterson his seat.

"I've never seen a qualified incumbent not be reappointed, and Rev. Peterson is not a 'quota' black. He was an educator, a leader in the community, an image to kids. . . There are so few people in the black community who really get out and try to do something, and then to get cut down for no reason whatsoever has got to be either racially or politically motivated," Greene said later.

Several leaders among Alexandria's black residents have complained recently of what they say is growing effort among the city's white office-holders to silence vocal blacks.

"Anyone who speaks out in the city of Alexandria, especially if they're black, seems to be reprimanded in the city, and I'm afraid it's taken over in terms of Peterson's role," said former board member Ferdinand Day yesterday. t

"It was quite a shock," agreed council member James Moran, who is white and who voted for Peterson. "But clearly there was a concerted effort to remove him on the part of some people in the community and some council members."

Moran, who started receiving anti-Peterson calls and letters about a month ago, said he was contacted by at least two school board members and a former board chairman who served with Peterson asking him not to support the reappointment.

"They said they were offended by his strong stance on racial issues . . . that everything was always a question of race," Moran said.

There is traditionally little contact between council members and the school board at reappointment time, Tyler said yesterday, except to offer support for a colleague. That some board members supported Peterson's removal, she said, may signal hard times ahead for blacks in the city.

"I find rising in Alexandria the idea that if a black person is strong and articulate and intelligent, particularly highly intelligent, that the forces come to the front to try to either weaken that person or get them out of the roles of leadership they are playing," she said.

Moran said one board member who contacted him complained that when actions were taken by the board that Peterson disagreed with, Peterson was "very quick to call it racist."

Peterson was out of the country yesterday on business and unavailable for comment, but board member Euille defended his colleague, saying any criticisms were based on misunderstandings.

"We all have racism characteristics. Its content and intent depend, however, on who's saying it and why," he said. "In terms of Rev. Peterson's role as a black leader and a leader in a community, his remarks were proper."

Mayor Charles Beatley, who also voted for Peterson, said yesterday that if personality conflicts were the reason for Peterson's defeat, then perhaps another tack toward appointment should have been taken.

"We've made a big mistake. We owe the black community substantial input into the school system," Beatley said. "If people objected to Peterson, then they should hve found someone else from the [black] community. There must have been other good people who could have replaced him."