Ben Vereen is a star. He's scaled all the walls of show business from television to Las Vegas, but now, at 35, he has come to what he calls a "change of direction."

He left Broadway on the wave created by "Pippin" in 1972, touring a cabaret act (which he brings to Wolf Trap tomorrow) and plugging into the star machine: the talk shows, the Oscar and Tony award presentations, the charity balls and telethons, the Miss Black America pageants and guest spots on TV specials, and his own series.

But after "All That Jazz," the film roles for vibrant young black men just haven't been there. The biggest role in years, Coalhouse Walker in "Ragtime," went to someone else, and his prime-time series "Tenspeed and Brown Shoe" was a casualty of 1980. The prizes have come--for Chicken George in "Roots" and for a television special--but the appeal of the Hollywood circuit has paled. He isn't being offered that many parts, and the ones he is offered "are not up to standard." He was asked to join the parade of names on a televised gala at Ronald Reagan's inauguration, but the honor faded when his act, a tribute to Bert Williams, a black minstrel performer, was edited for television and provoked a storm of complaints from blacks who saw only the shufflin' and jivin' and a painted-on black face.

So recently he moved his wife, four daughters and one son back to the East Coast, to Saddle River, N.J., an upper-class enclave also known as the residence of Richard M. Nixon. "They asked us to move in to improve the neighborhood," he jokes. It's only 40 minutes from New York.

And four weeks ago he started dancing again, taking classes from May O'Donnell, a teacher he knew at the High School of Performing Arts, feeling the pain in his muscles and loving it.

"Even my hair hurts," he said. "But the pain is good because those muscles are coming back alive.

"It's a new lease on life. I had not been dancing the way I know dancing to be. I was doing commercial dancing. This is modern dancing, it has meaning and story and spirit and ritual. I had a need to rediscover myself, to get back to my roots."

"You're a star," says the TV host with that intense enthusiasm they all profess. "Where does all that talent come from?"

It's hard to know what the host expects as an answer. Where does talent come from? From his second cousin on his mother's side? From a sprinkle of fairy dust in his cradle? Vereen credits the Lord.

Before the show, his public relations woman has gone to the host and asked him not to mention "the inaugural thing." The request, of course, is ignored. She repeats it to the next reporter. "I just don't want him aggravated," she says. During the interview she sits nearby, taking notes.

"I was angry, and I was hurt. I was hurt because they should have known that I would never do anything to hurt black people," he said evenly when asked about the inaugural. "I was trying to make a statement about the suffrage of black people." It was the television director who did him in, he said, cutting his act so that the introduction, in which he explained how Bert Williams wasn't allowed on a stage unless he blacked up, was eliminated, and all the audience at home saw was an image of a grinning, shuffling stereoptype. "The Reagan cuts started that night," he joked half-heartedly.

"The Reagans sent two beautiful letters because they realized I got a lot of feedback. They got the message."

It seems you just can't do anything once you're a star without getting a lot of flak. In 1981 he was booed in Atlanta (along with former U.N. Ambassador Andrew Young) when he participated in a memorial march to the children who had been killed there; the crowd was frustrated with people who were talking about peace and unity. He canceled a trip to South Africa in 1981 under political pressure from South African blacks who think that performances by American entertainers lend tacit approval to apartheid policies.

Vereen's particular gift is the ability to fuse energy with skill, to create a kind of superb tension by juggling and meshing both song and dance with apparent equal facility. He is one of those entertainers who seems to be most nourished on stage, taut with the immediacy of performance and the intensity of the physical demands. His gestures are deft and clean, his voice as sinuous as his body.

"The higher the audience gets, the higher I get," he said. At first, film was hard for him. In "All That Jazz," he pretended that the grips and technicians were his audience. "They had a hard time with me at first because I was used to live energy. On stage you can stretch out; in the movies you have to hit that mark."

Offstage, as he was on a recent stopover here, he seems tense with the self-consciousness of a man eager to please but uncomfortable being interviewed. At regular intervals he flashes his 150-watt grin and pops a bon mot, a sure laugh-getter, an audience pleaser.

"I was raised in what we call the Upper Ghetto," he said. "You mean Harlem?" said the talk show host. "No," he said. "We ate."

Vereen was corraled into dancing school by a man soliciting students off the streets of Brooklyn. He bent Vereen's legs this way and that and told his mother the kid had great potential. "It was the Startime Dance Studio. I remember the teacher reading steps from a Fred Astaire Dance Studio book."

Later the principal of his junior high, P.S. 178, recommended him to the prestigious High School of Performing Arts. Students are admitted only after an audition, and Vereen remembers dancing to Quincy Jones' "Killer Joe," wearing Bermuda shorts, a T-shirt and sneakers. The reviewing panel included Martha Graham, George Balanchine and Jerome Robbins, but because Vereen didn't know who they were, he wasn't intimidated. "I really want to get into this school," he said to a friendly looking woman. "Don't worry," she said. She turned out to be the head of the dance department, and he did get in, turning up for the first day of school dressed in a collarless sharkskin suit and carrying an attache' case. He hadn't heard about tights and dance bags yet.

"I was lucky, they needed boys," he says now. But of the 10 in his class, only four made it through the first-year cut.

"I never thought about success," he said. "I used to tell my Mom: 'I just want to be on the totem pole.' "

Now he is on the school's list of VIP graduates, and he remains loyal --although he did not like the cinematization of it in the movie "Fame."

"We would never have come to class that scruffy," he said. "P.A. has a certain elegance and dignity about it."

After he graduated, he toyed briefly with the idea of becoming a preacher like his grandfather, but this moment of youthful quandary was soon assuaged. "A minister said to me, 'The Lord has given you a gift. If you don't use it, he'll take it away.'"

He wants to use it next in a musical. Back to Broadway--once he finds the right material. "There's enough revivals and revues on now," he said. "I'm looking for a real theatrical piece, an 'Oklahoma,' a 'Guys and Dolls.' "

He also wants to open a dancing school, not a Startime studio but a serious place to study modern dance. "When I want a suit, I don't go to a department store, I go to a tailor. When I want shoes, I go to a cobbler, not a shoe store. There's a lot of bad schools out there taking people's money. Besides, I need a place to go to myself."