IIie Nastase, whose genius is so often overwhelmed by the eccentricities that go with it, was beaten in the second round of the U.S. Open tennis championships today, 6-4, 6-4, by Italian Davis Cup player Corrado Barazzutti.

This was a typical Nastase performance. At the start, the mercurial Romanian was all touch and artistry. But suddenly, inexplicably, his game sourced and he turned surly. Every distraction preyed on him. The crowd egged him on, and he swore and gestured profinely at his real and imagined tormentors.

He cursed tthe grip on his racket, which slipped in his hand several times on a beastly hot and humid day. He snarled at photographers, linemen, spectators and at his manager-confidant, Mitch Opera. Finally, his wife left her courtside box, sobbing quietly.

At the end, after Barazzutti has choked away a chance to get to 5-3 in the second set with two double faults. Nastase quit. After losing his serve again, he played the final, game half-heartedly as the crowd showered him with boos and catcalls.

Nastase's collapse in his familiar muttering, sputtering jumble of nerves was the beginning of a wild day at the West Side Tennis Club, where 12.061 matinee spectators seemed to sweat as much as the players who kept 12 courts throbbing with activity. Some of the other highlights:

Billie Jean King, the four-time championwho was playing her first singles match here since beating Evonne Goolagong in the 1974 final, blew a 5-2 lead in the final set but finally shook off 17-year-old Anne Smith, 6-3, 3-6, 7-5, in an excruciating match.

Mike Fishbach, 22, an amply beared, amusing, apple juice-slugging refugee from the satellite circuit, used a racket that he has strung in a revoluntary grid with two interwoven layers of gut reinforced with fish test line, adhesive tape and twine, to wallop No. 16 seed Stan Smith, 6-0, 6-2.

Fishbach strings the rackets himself, using a variation on theme developed by a German engineer and first used in a major tournament by Australian Barry Phillips-Moore in this year's French Open. He puts two sets of strings in a conventional racket head (the process took him 19 hours the first time, four hours now), and compares his racket to a sponge table tennis bat, which puts exaggerated topspin on the ball.

"On a good day, my ground-strokes have very heavy top, they go high over the net, dip sharply, and bounce up very high to the opponent with this racket," said Fishbach. "It's got 14 up-and-down strings on both sides, with six rows of crosses going in between the up-and-downs which sandwich the crosses. It's like a joke to look at.

The racket has given Fishbach, an engagingly good-humored felloe who quit tennis foe six months last year and went to live in the northernmost Green Mountains of New Hampshire, asurge of confidence in tight situations where he admittedly used to crack. Smith's lethargy also helped him today. "He was very lackadaisical out there . . . I played a pretty consistent game, and Stan had a very off-day," Fishbach said.

Bjorn Borg, the Wimbledon champion the past two years and No. 1 seed, beat Australian second-liner John James, 7-5, 6-4, but said that his injured right shoulder is increasingly tender and could still force him to defualt.

Borg again served at only half pace, was noticeably tentative overhead, and said that for the first time the damaged muscle in his upper right chest hurt him on groundstrokes. Borg said he would not practice Saturday and would have to test the shoulder before deciding whether to play Onny Parum in his scheduled third-round match Sunday.

Ken Rosewall, 42, gave fellow Australian Phil Dent, 27, a semifinalists in the Italian and French opens this year, a lesson in compact, efficient shotmaking. "I thought I played pretty well and I got four games," shruggedDent, who seemed slightly shellshocked by his 6-1, 6-3, defeat at the hands of a man who is truly a precious antique.

Chris Evert, in a rare appearance on a side court, decided not to stay there long. The champion of 1975-76, seeded No. 1, took only 36 minutes to blitz Australian Pam Whytcross, 6-0, 6-0, for her 108th consecutive victory on clay courts. Evert, who has won 22 straight tournaments on her favorite surface, dating back to August, 1973 (before the U.S. Open shifted from grass to artificial clay), won 51 of 69 points in the massacre.

Brain Gottfried, the No. 3 seed, blasted and lobbed his way out of a l5 deficit in the first set, beat Marty Riessen, 7-5, 2-6, 6-3, and then was overcome with stomach cramps.

No. 10 seed Dick Stockton, tow points from elimination as he served at 5-6, 0-30 in the final set, recovered and won a tie breaker, 7 points to 3 to complete a 3-6, 6-3, 7-6 victory over John Alexander. This was a splended and hard-fought match between two attacking players who have learned to temper their aggression and play patiently on clay.

The other seeds who played this afternoon all advanced: No. 5 Manuel Orantes over Fred Stolle, 6-2, 6-2: No. 9 Eddie Dibbs over Fred McNair IV of Chevy Chase, Md., 6-2, 6-2: No. 11 Roscoe Tanner over Andrew Pattison, 6-7, 6-0, 6-3: No. 12 Harold Solomon of Silver Spring, Md., over Paul Gerken, 6-3, 6-1; and No. 15 Wojtek Fibak over Sashi Menon, 6-1, 6-2.

In women's singles, No. 4 Sue Barker moved into the round of 16 with a 6-4, 4-6, 6-1 victory over Iris Riedel, and Wimbledon runner-up Betty Stove, seeded No. 5, disposed of lefty IIana Kloss, 6-4, 6-1.

Nastase, the champion here in 1972 and No. 7 seed this year, did not lose a game in his first-round match against Frew McMillan and started in a sprint today. He led, 3-0, in the first set and had an advantage point for 4-0, but then everything crumbled.

Barazzutti, a formidable, industrious clay-court player who has worked himself into great shape for Italy's Davis Cup semifinal series against France the weekend after next, settled down into a pattern of hitting deep groundstrokes and passing Nastase when the Romanian tried to force the attack.