Guillermo Vilas, looking sometimes like the champion he was a year ago and at other times like an indifferent tennis player with more weighty subject on his mind, struggled to a torturous 5-7, 6-4 victory over Texan Bill Scanlon, the 1976 NCAA champion, in the second round of the U.S. Open yesterday.

It was a match that characterized Vila's 1978 - a season that has been, in his words, "not great but not bad . . . I haven't played that well, but I don't think I've played below my class."

Now and again the astounding shots of the vintage 1977 Vilas were there. The sprints and flashing winners from defensive position. Topspin backhands buzzed a cross court on the dead run. Lob winners launched with the flick of a singularly strong wrist.

They made 14,000 sectators at the new National Tennis Center in Flushing Meadow Park forget that Vilas does not like the rubberized asphalt surface that replaced his beloved clay when America's premier tournament moved from its home of 54 years, the West Side Tennis Club in Forest Hills.

The piece de resistence was a wondeful, saying half-volley, hit at full stretch off a ball already behind him. It came off what looked to be a winning forehand return on Scanlon's third advantage point to level at 4-4 in the final set.

After that Scanlon, a talented 21-year-old who has not lived up to the rich promises he showed in 1976-77, won only one more point.

That astonishing half-volley - the embodiment of reflexes, touch, and athleticism - was a stroke that cried out something most people here seem to forget: "I, Guillermo Vilas, am the defending champion." No Bjorn Borg, who defeaulted in the fourth round last year with a bum shoulder. Not Jimmy Connors, whom Vilas beat in a bruising final, 2-6, 6-3, 7-6, 6-9.

But at other times, Vilas did not look at all like he did last year, the poet who became a killer at age 25. He was the anthesis of sharpness, squandering opportunities to blow Scanlon away.

Vilas came back from 2-5 to 5-5 in the first set - as he had in beating Cliff Drysdale, 7-5, 6-2 in the first round - then lost the set with a couple of terribly ragged forehands.

In both the second and third sets, he got up a break and promptly lost his serve at love. He got up another break in the third, but let Scanlon get to advantage twice in the sixth game and three times in the eighth, and was rather lucky to escape.

"I'd have to say he's vulnerable, because I honestly think I could have won that match," said Scanlon.

Certainly Vilas is not playing up to his standard of 1977, when he won the French and U.S. opens, captured the $300,000 first prize for topping the Colgate Grand Prix standings and rolled up a 50-match winning streak in a punishing July-through October rampage. He battled Borg and Connors for the No.1 world ranking, and the three of them were in a class apart, a lofty plateau above everyone else in the game.

To do this required a supreme physical and mental effort from the left-hander from Mar del Plata, Argentina, Vilaswas driven. He put himself in the hands of his coach-manager, Romanian Ion Tiriac, and worked harder than anyone else, sweating through uncompromising four to-six hour training sessions, flogging himself into peak fitness.

He put aside the other interests in his life: poetry, music, Eastern philosophy, a catalogue of intellectual self-improvement projects, a sreenplay on which he was working.

That is not his nature; Vilas' horizons will never be as narrow as the dimensions of a tennis court, and to deny his other passions indefinitely would constitute waking up in the morning and spitting on himself.

Vilas says his motivation, his desire to excel in tennis, has not dimishe."I am always interested. If I did not want to win, I would not be here. The writing, the other things, they are all on the side. I do not mix them with tennis expect when I am stupid and forget it is impossible."

Tiriac tells a different tale, however. Vilas completed his second book, to be published at the end of the year, the day he lost to young New Zealander Chris Lewis in Kitzbuhel, Austira, he points out. Vilas is preoccupied with "Players," the movie in which he stars with Ali McGraw and Dean Martin Jr., the coach says.Vilas gives the time to his workouts - still upward of four hours a day - but the intensity is not there.

"I try to get him to devote himself to tennis, but it is his life and he does with it what he wants. I cannot change him," said Tivlac, "Guillermo is a special person."