"This is pretty fancy stuff," said Benny Goodman, looking around the Benjamin Franklin Room of the State Department. There, 250 of the nation's most artistically talented and politically powerful gathered Saturday night to pay tribute to Goodman, three other kings and one queen of America's performing arts as part of the Kennedy Center's fifth annual honors ceremony.

"I don't know the politicians, I lay low there," said clarinetist Goodman, 73, father of the swinging "big band" sound. Broadway producer George Abbott, another honoree, sprightly at 95 and with a revival of "On Your Toes" about to open at the Kennedy Center, said that after a lifetime of honors, "It'd be pretty hard to top this one. In fact, I don't expect to."

It was a weekend-long celebration of the performing arts. Saturday night the honorees were officially presented with their award medallions at a dinner hosted by the Kennedy Center Board of Trustees. Yesterday, they went to the White House for a late-afternoon reception with the Reagans, and last night it was on to the Kennedy Center for a gala performance, which will be taped for television, and a black-tie dinner.

Even for Washington, it was an amazing constellation of theatrical and political stars gathering at one time in the same place.

At the State Department Saturday night, there was, for example:

Walter Mondale, not a hair out of place, looking presidential and, when asked, praising Ted Kennedy's "tough, courageous decision" to get out of the 1984 race. He then laughed when Sen. James A. McClure (R-Idaho) cuffed him on the arm and said, "You look like you're mean and in fighting trim! "

Yves Montand, wearing a "Solidarnosc" button given him by Lech Walesa when the Polish labor leader visited Paris. Montand is saying "bonjour" and "enchante" to people in that gentleman-macho way of his that has sent many a chill down many a spine.

Playwright Edward Albee, dressed in a tux but managing to look suitably seedy anyway, casually leaning on the rail of the eighth-floor balcony and looking out over the twinkling city and the nation's monuments bathed in pools of light, looking for the new Vietnam Veterans Memorial.

"Why isn't it lighted?" he asks.

And Cary Grant, gray hair, tan, black-frame glasses, who received a Kennedy Center honor last year. Looking just . . . splendid. With his wife Barbara, looking equally splendid. The press photographers went bananas when the Grants walked into the State Department's C Street entrance. Flashbulbs galore.

Ethel Kennedy slipped in behind them, almost without being noticed.

"Gosh, we walked into a good elevator," she said, smiling at Grant.

"Meet my wife, Barbara," said Grant.

"It's been a long time," said Kennedy. "He hasn't changed. It isn't fair. Make him do the cooking and the cleaning."

"Oh, he does," said Barbara Grant.

Later in the evening, after the cocktail party and the dinner honoring the five greats -- the others were Lillian Gish, star of "Birth of a Nation," dancer and movie star Gene Kelly, and conductor laureate of the Philadelphia Orchestra Eugene Ormandy--Grant hammed it up with Eva Marie Saint, who looked just about as sizzling as when she and Grant played that memorable train dining car scene in Hitchcock's "North by Northwest." Saturday night, they kissed and teased each other in the State Department entrance hall in front of a sign that said, "Welcome President Zia," who arrives today from Pakistan for an official visit.

Ethel Kennedy said her brother-in-law's decision last week was "right. Sad for the country but wonderful for all of us who love him."

"Oh, Ethel darling," said Sen. Charles Percy (R-Ill.). "I was so upset . . . " They turned aside for a brief private talk.

Douglas Fairbanks Jr. was one of the first to arrive. He was very tanned.

Elizabeth Fulbright came in before her husband, former senator J. William Fulbright. It was 70 degrees outside and she said, "What a night! You don't need any furs."

When Librarian of Congress Daniel Boorstin and his wife, Ruth, arrived, she said, "It's probably freezing in California."

The gray and black limos were arriving apace. Singer Peggy Lee accompanied by Frederick Goodson. Montand accompanied by Jane Herman of the Metropolitan Opera. Walter and Betsy Cronkite. Melvin and Barbara Laird. Actor Eddie Albert. Kennedy Center Chairman Roger L. Stevens.

Gene Kelly, looking young and athletic at 70, came with his son, Tim, a college student, and two daughters, Bridget and Kerry. "It's the whole crew," he said. " . . . I'm overwhelmed and overjoyed."

Gish, 86, wore a cream-colored gown and cream fur coat. She looked somewhat frail but her beauty was there, almost ethereal now. She arrived with Brooke Astor, a New York philanthropist.

"It makes you feel so inadequate because you can't give back in kind," Gish said of the honor. Then, unasked, she told a story. "You know, when Mary Pickford went to Russia in '28, she came back and said, 'You shouldn't go to Russia! You couldn't stand all that!' You see, they don't get people from the outside, so they kiss you and kiss you! "

Ormandy, for 44 years the conductor of the Philadelphia Orchestra and the author of its famed rich and romantic sound, is a very short man who retains, at 83, a good bit of the enormous physical presence and energy for which he was known. "It's marvelous, a great experience," he said simply of the honor.

Attorney General William French Smith and his wife Jean came in just before Gerald and Eden Rafshoon. The Smiths strode in briskly, deftly avoiding the photographers and reporters, clearly knowing, unlike many others there, exactly how to get to the eighth floor of the State Department.

Sen. Percy came in looking for his wife, Loraine. He had just flown into town and hadn't had time to pick her up. "I just came in from Springfield," he said in that deep, senatorial voice of his. "I changed clothes in the middle of the airplane." Pause. "Not in the middle of the aisle, of course." He laughed.

At the cocktail party, Montand was on the balcony saying to Herman, "On travaille comme un chien . . . " Montand said he came because Gene Kelly asked him personally to take part in last night's gala show, reviewing the lives and accomplishments of the five honorees. The gala will become a two-hour TV special, produced by George Stevens Jr. and Nick Vanoff, to be shown nationally by CBS on Christmas from 9 to 11 p.m.

Composer Richard Adler came up to Montand, shook his hand, and spoke rapidly and excitedly about several things. "And we have the same barber, Rocco!" said Adler. Then he rushed off.

Asked about the significance of the honors, Albee said, "It is interesting to me that we have been able to support the arts despite what the Reagan administration has been trying to do . . . Our government has a responsibility to educate people esthetically."

Nearby was Frank Hodsoll, the former high-ranking White House aide who a year ago was named chairman of the National Endowment for the Arts. Asked to respond to Albee's remark, he said, "I think we're doing a fair amount for the arts. The president is committed to the national government having an important role."

While the budgets for the NEA and its sister National Endowment for the Humanities have not been slashed under Reagan, they have leveled off and the administration is emphasizing private donations, rather than federal grants, to support the arts.

Mondale arrived with his wife Joan, who was chairman of the Federal Council on the Arts and Humanities during the Carter administration. Of Ted Kennedy's decision, Mondale further said: "I think he handled it elegantly. I think it's going to enhance his influence in our party and in the country."

After a dinner of roast quail and dessert of florentine nests with fresh strawberries and zabaglione, the medallions for excellence in the performing arts were presented to each honoree in a moving ceremony emceed by Grant. Then each honoree was toasted by a friend from the often-distant past.

Acting Secretary of State Kenneth Dam called the honorees "five of the most sparkling jewels in our national crown . . . The arts are really our best diplomacy." Dam hosted the dinner in place of Secretary of State George Shultz, who was with the president on his way back from South America.

Laird, speaking briefly at the ceremony, said, "The Kennedy Center has truly become the national center for the performing arts."

Grant's introductions for each artist were pithy, at times funny and moving. He called Abbott "Mr. Broadway," a Renaissance man of the theater who "can write a lyric, fix a script, choreograph a dance." Jean Stapleton toasted Abbott and recalled those days of working on "Damn Yankees" when, "We called him Mr. Abbott because he was stern, but he had a twinkle in his eye and, when he smiled, it was sunshine.

"You demanded of us energy . . . and, above all, joy, and we all thank you, George."

Of Gish, Grant said, "Her life story is part of American mythology. She and D. W. Griffith created the American cinema as we know it." George Stevens Jr., head of the American Film Institute, toasted Gish movingly and then raised his glass to "Miss Lillian," as Griffith always called her, and the crowd stood silently for a minute as everyone raised their glasses in tribute.

Grant described Benny Goodman as the "son of a Polish immigrant" and remembered how he and his band had given the first jazz concert ever at Carnegie Hall. Then there was a standing ovation for the man who toasted Goodman. He was Lionel Hampton, a black vibraharp player hired by Goodman in 1936. "Benny hired two black musicians and put them in his orchestra," said Hampton. " . . .It made a great change in our country because, before, black and white never appeared on the stage together. I asked him, 'What's your idea in hiring two black men?' and he said, 'Man, on the piano we need the white keys and the black keys.' "

Gene Kelly, Grant said, started dancing to "impress the girls" and went on until his name "became a synonym for the grace and drama in American dance . . . It was with Gene Kelly standing next to her that Mary Martin first sang, 'My Heart Belongs to Daddy.' " Betty Comden and Adolph Green, who among their many collaborations wrote the script for "Singin' in the Rain," got laughs in a duet toast to "that great Renaissance man from Pittsburgh!"

Ormandy, Grant said, was the creator of "the Philadelphia sound" and a man who had the scores of the world's great music "in his head and deep in his heart." Pianist Van Cliburn then toasted Ormandy. "His ideas regarding music are Olympian," Cliburn said. " . . . So much fire, so much dedication . . . How much your music has given me spiritual consolation at times when I needed it so desperately!"