George McGovern arrived 20 minutes early. Ted Kennedy was 10 minutes early. And Elizabeth Taylor and John Warner, who were a little late, had to cool their heels in their limousine while waiting to get in.

It was Lorraine and former Kentucky Sen. John Sherman Cooper's 16th annual garden party for the United States Senate Thursday night, and as one of the nearly 400 invited guests, former OAS Ambassador Sol Linowitz, put it: "So who's left back in town?"

The famous names and faces lined up on N Street NW before the front door of the Coopers' Georgetown residence supplied vivid evidence of the place to be in Washington Thursday night.

"I always think this is the nicest event in the springtime and I always think it's because of John," said Kennedy, explaining that Cooper is on the advisory committee of the Kennedy School of Government at Harvard "and was a special friend of President Kennedy." In fact, one guest recalled, Cooper, a Republican, and John F. Kennedy, when he was a senator, co-sponsored the first Senate resolution establishing the concept of foreign aid for developing countries. In those days, the John F. Kennedys and the John Sherman Coopers, despite their stated political allegiances, were thick as fleas.

Yesterday, as an indication that old loyalties remain, Sen. Edward M. Kennedy rushed back from Massachusetts where he attended nephew John's graduation.

"I'm going to come over and have lunch with you and not tell your husband about it," Kennedy flirtatiously promised Lorraine Cooper.

Mrs. Cooper was wearing a blue and white St. Laurent cotton plaid, a gown whose ruffled hem swept daringly up one side to reveal a trim and provocative knee. "It drives my maid crazy." she confided, with a coquettish lift of an eyebrow. "I'm an older woman, you see."

Standing with her husband in their elegantly appointed drawing room, she flicked a colored straw fan as much to stave off kisses as to circulate the warm evening air.

"Don't kiss me - it's too hot," she told several guests who approached with puckered lips.

Even so, there were some who managed to slip past the fan, among them Ambassador Ellsworth Bunker and British Ambassador Peter Jay.

The guest list was more than just bipartisan. It included faces from the old Roosevelt New Deal. When Ben Cohen turned up, Lorraine Cooper told her husband he needed no introduction.

"I knew him before you did. We went steady even before you came to Washington."

Back in the beginning when they first started giving the party, the Coopers restricted their guest list to Republicans.

"Then we thought 'what the devil', said Cooper, "and we decided to open it up. I've gotten over labels like conservatives and moderates. They all vote differently anyway. You can't tell."

The formula proved to be a winner and through the years, few of the Cooper's invited guests ever sent regrets. But even this year's turnout seemed to be some kind of a record, according to the Coopers' butler Michael.

"Bigger even than the Bicentennial party we gave in East Germany" (where Cooper was U.S. ambassador).

In the garden, where tables were set for 230 diners at any given time ("it's rather like opening a restaurant for an evening," said Tom Hanson, who designs Mrs. Cooper's parties), people were trapped in the aisles, unable because of the crush to move toward the buffet table and its chicken a l'orange but more critically, toward the bar.

Devron's band played a little night music for anyone able to reach the dance floor. Seated around it nearly (but not quite) unnoticed were the two senators from Virginia with their wives (who soon slipped out through the kitchen door).

Retiring Gen. Alexander Haig, at another table, said his Washington return was "only temporary. No, I'm not overwhelmed by the pressure" to seek the presidency, he said.

Showing off his new bride to his former colleagues was Ed Brooke, making a rare appearance on the Georgetown party circuit.

Elsewhere in the crowd were multi-millionaire Joe Allbritton, saying he was making a little more money - "and I need it" - running Channel 7 but not having as much fun as when he owned The Washington Star.

Attorney General Griffin Bell spotted South Carolina Sen. Strom Thurmond, of the Judiciary Committee, and over the head of Chief Justice Warren Burger, let Thurmond know he's nominating "a deputy AG you'll be quite pleased with."

Eventually Lorraine Cooper joined the group in the garden.

"She loves parties, loves attractive places and people. If they aren't attractive enough," said one guest, "well, there's plenty of champagne so they'll look attractive by the end of the evening."