Critics call Walter Williams the most conservative black in America -- a label the George Mason University economist decries as untrue. But Williams, a man who disapproves of affirmative action and most civil rights groups, has a lot of people in Ronald Reagan's Washington listening to him.

To explain himself, the 45-year-old professor says he has always looked at things from a different angle. It goes back to youth in the North Philadelphia ghettos, he says, when the upset his family, devoted churchgoers, by telling them he didn't believe everything he read in the Bible.

As an Army recruit more than 20 years ago, Williams continued to make waves. He antagonized officials when he persuaded a group of fellow black soldiers at Fort Stewart, Ga., to crash a white soldiers' dance there, refused to march in a St. Patrick's Day parade and later, after being shipped to Korea, filled out a military form saying he was "Caucasian."

Even as an economics professor at Temple University in 1974, Williams didn't escape controversy. He angered a group of black students when he sent out a memo (later given to a local newspaper) saying some white professors at the Philadelphia school were giving black students easy grades, something he called a form of racism.

Today, Williams is a respected and widely known Washington economist. He was part of the Ronald Reagan transition team and he is an associate of Nobel laureate Milton Friedman and a member of the Mont Pelerin Society, an international economic group.

But his critics are numerous. The tag they put on him -- of being America's most conservative black, -- gets this response: "I recognize the need of people to use labels, but I don't consider myself a conservative.

"If you have to use a label for me, then I'm radical," he said recently in his small, third-floor university office in Fairfax County. "I've always been radical. I guess I'm just one of those crazy Americans who believes in individual freedom."

Despite his dislike for the black conservative label, Williams, who calls himself a provocative scholar, has risen to national prominence with it.

Though national television appearances (he is on the Phil Donahue Show in Washington this morning), radio talk shows, speaking engagements at conventions and through a syndicated "Minority Viewpoint" column he writes for 39 newspapers, Williams has been preaching a gospel for solving the problems of America's blacks and poor that is miles away from traditional ideas offered by civil rights activists since the 1960s.

Though he acknowledges he was "raised on and off welfare," he rejects many social programs as ill-conceived. He is against affirmative action, he decries the national minimum wage laws, believes busing is unfair and deplores the alliance between American blacks and the labor unions.

Williams, striking a casual pose with his feet propped high on his desk and cigarette in hand, says he doesn't mind the stinging criticism he's gotten from civil rights leaders: "I don't give a damn whether I'm in the majority or not. I just try to look at everything in an analytical manner."

Williams says his assessment of the problems of black America is more accurate than that of civil rights leader -- who he says represent only middle-class blacks -- because his statements are based on careful study and analysis.

He contends affirmative action programs hurt blacks because some unqualified blacks and getting into professions. "We, as black people, can't afford incompetence. I can understand the impatience of people wanting to see blacks in these positions now, but what's more effective and more permanent?"

The national minimum wage laws, he says, put many teen-agers, particularly disadvantaged ones, out of work: "The effects of not allowing less preferred people to charge a lower price [for working] spells less employment for them."

Williams, who says he sent his 6-year-old daughter to school at 18 months, believes everyone should have the free choice to attend school whereever he pleases. "We have to put lower in the hands of the parent."

And he accuses civil rights groups of being "in bed with the labor unions," which he says aren't in the best interest of blacks.

And he accuses civil rights groups of being "in bed with the labor unions, which he says aren't in the best interest of blacks.

Carl Rowan, a noted black journalist, dismisses Williams as "a phony and a jerk," a class of person he says he holds in "utter contempt." Rowan says Williams is particularly controversial with other blacks because he is against social programs that have been helpful to many poor blacks.

Williams, says Rowan, has views parallel to America's conservative whites and is "one of those persons who's helping the Reagan administration take food out of the mouths of black kids. . . ."

Vernon Jordan, director of the National Urban League, calls Williams "a man without a constituency." Michael Meyers, assistant director of the NAACP, says Williams is "a reactionary conservative" who is opposing "the very channels responsible for black people being in the mainstream of American life."

"That doesn't bother me," says William. "It's just room for more encouragement. Black people have a diversity of interests. A lot of things [civil rights group] support are not in the best interests of blacks."

Las month, after spending a year as a visiting professor at George Mason, Williams joined the faculty as one of 22 full-time economics professors.

Williams finds himself on the defenseive at the rapidly growing, state-run university. Several black officials there are questioning the school's motive for bringing him to the faculty.

Frank Matthews, director of the school's affirmative action office, says he's worried because Williams' appointment "will give mixed signals to the casual observer."

But Philip Coelho, newly named chairman of George Mason's economics department, says he doesn't think Williams' political beliefs have anything to do with his hiring. ". . . He's a very competent and fine economist. We may have a number of people who are in agreement with him on certain issues, but it's not a department policy to have a certain kind of philosophy," Coelho said.

George Johnson, president of the university, discounts theories that Williams was recruited to help the school project a conservative image. "That's ridiculous. We're very happy to have him at George Mason because he's a very independt-minded man."

Williams, who received hi PhD from UCLA in 1972, disputes claims that he has not always been independently minded and was heavily influenced by Thomas Sowell, another black conservative economist.

"I met [Sowell] at UCLA. He was interviewing and I was in the process of leaving," Williams said. "Never had a clas with him, but we got together a lot and discussed a lot of black problems.

"I've always been an individualist. As a matter of fact, [Sowell] might consider me more extreme than he. Look at some of my earlier papers and columns and you'll see that I haven't changed."

He says no one can link him to a particular cause, although he was part of the Reagan transition team. "I don't vote. I haven't voted since 1964. It doesn't pay me to vote. The chances of my vote counting out of all those votes is minus 10 to the 10,000th [power]."

Williams says part of the reason he's been controversial is because of his questioning nature.

"I don't go out of my way to be different. . . . I've just got my own mind. I've always been a questioner," Williams said. "I didn't accept things because they were there."