Legitimacy is hard to come by in the U.S. Football League, the league with a penchant for controversy. And it might be hardest for Herschel Walker, its first star, who came cartwheeling out of college with a little controversy of his own.

Walker, who departed Georgia after three years with an armload of dollars and unfulfilled potential, rushed for 2,411 yards in 18 games with the New Jersey Generals this season and, in 16 games, broke Eric Dickerson's all-time single-season record of 2,105 yards, set in 16 games with the Los Angeles Rams last season.

Dickerson's response was typical of the general response to Walker since he gave up the glory of the college game for the USFL, forgoing his senior season. Dickerson said he didn't think Walker's mark could be compared to his because it was set in the "minors."

It is a thought that is widely held. With skepticism still rampant despite his record, Walker seems to have reached an impasse with his critics.

"Everyone is entitled to their opinion," Walker said. "A minor league? If I'm being paid this amount of money to play in a minor league, then I guess I'll be a semipro all my life."

Controversy is a strange state of affairs to a man whose ambition is to be a G-man or a police officer and who once pulled a woman from a car wreck by tearing off the door. Walker's informal design for living has something to do with good works and duty and those sorts of things.

The quiet, good life of Walker includes a Mercedes-Benz and a seven-room high-rise apartment in the New York area, his only concessions to luxury and high living. A 23-year-old from the small and seriously Baptist town of Wrightsville, Ga., Walker chooses to ignore his wealth. It is no use asking what it is he does with all that money, because other than give it away, he doesn't do anything with the bothersome stuff.

"I save it."

For what?

"Nothing really. It just seems silly to throw it away."

Some of Walker's money goes to charity; he gave Georgia $100,000 last year for a new athletic building. The rest goes into savings or his two companies, a construction business and a fast-food franchise.

When rookie quarterback Doug Flutie signed a contract with New Jersey for an estimated $1.25 million a year, owner Donald Trump extended Walker's contract a year and raised his salary to a reported $1 million.

"The money doesn't seem to affect him," his coach, Walt Michaels, said. "That he gets paid is incidental to him. You've got to love the game first, or it gets awfully dull. That's his biggest plus."

The question then arises: what does Walker do for fun? He spends his offseasons traveling with his wife Cindy to various speaking engagements and charity functions. Or he simply stays at home being quiet.

"I don't know," Generals fullback Maurice Carthon said. "I'll call him up in the offseason and say, 'What are you doing?' He says, 'Oh, just sitting around.' "

"I'll tell you what he does," said Gil Brandt, Dallas Cowboys director of personnel development. "He runs. That's what he likes best."

The suspicion that Walker might have been complacent in his first pro seasons because of his contract now seems less likely in light of his extravagant accomplishments.

Walker was criticized for those first two seasons with the Generals, which were more than respectable but not the dazzling successes expected. He gained 1,812 yards his rookie season, but it wasn't until the fifth game that he had 100 yards in one contest. Last year he gained 1,339 yards, and the talk around the league said he wasn't the punishing runner he had been in winning the 1982 Heisman Trophy.

Walker, who doesn't like to be called punishing, anyway, because it doesn't sound very artistic, attributes his spectacular season to some newfound good health. He says that for the first time since he left Georgia, he hasn't been troubled by a sore shoulder, an operation in the offseason to tighten ligaments curing the injury.

"I just let people draw their own conclusions when I went in for surgery," he said. "I found out in preseason last year that I needed it, but I wanted to go through the season. What people say doesn't bother me. People resent the USFL, and resent me being in it. The only way for me to be the best is to listen to myself.

"When you're in the public eye, people are always more knowledgeable about your life than you are. Coming from a small town, I learned to ignore the talk and go on about my life."

If there is resentment toward Walker, it probably is mixed with a sense of loss because perhaps the greatest college back of all time did not stick around for his senior season.

"He was always overly praised, and overly criticized," Georgia Coach Vince Dooley said. "I guess that's the nature of the business."

Yet Dooley's eyes still fill at the thought of what Walker might have accomplished with the Bulldogs. At the end of his junior season, he had 5,259 yards, third on the all-time NCAA list, and was just 823 yards from Tony Dorsett's rushing record.

"He could have set records that never would have been broken," Dooley said. "He would have been the all-time college back. He could have gotten Dorsett's record by the fourth game."

Dooley, who keeps in touch with his most famous player, maintains that Walker was a reluctant pro.

"I really don't think he wanted to do it," Dooley said. "Right there at the end I think he wanted to get out. He enjoyed trying to find out what he was worth, got too far in it and couldn't get out. I told him to go forward and make the best of it. From his standpoint, I wish it had been otherwise.

"He was held in such high regard, he was almost too good to be true. He was a hero to everyone in every respect. He won the Heisman, and right after that, he saved a woman from a car wreck, like Superman or something. Then the controversy hit. So I think it tarnished his image a little bit."

Walker's position from the beginning has been that he does not regret his decision. "I never thought about it," he said. "I'm ahead of a lot of 23-year-olds, individually and as an athlete. I got a two-year head start on myself."

Respect might not be far behind. Walker regained another measure of legitimacy when the Cowboys selected him in the fifth round in this year's NFL draft, although what that might mean for his future isn't certain since his contract with the Generals runs through 1988.

"Only time will tell if he'll do it in the NFL," said the Cowboys' Brandt, who has become one of Walker's few close friends. "I think he will."

Brandt says Walker is "content being low key." Walker's coach, Michaels, puts it a different way.

"People don't want to admit he can play," Michaels said. "Don't worry. He can play."