They behaved like back-yard neighbors: chummy, chatty and, as the weekend wore on, occasionally chilly. They were the leaders of the free world together at President Reagan's Memorial Day house party in this historic and incredibly quaint 18th-century setting.

Sometimes, they weren't too different from any other tourists, except that this "back yard" is the ultimate of exclusivity.

But despite the omnipresence of hundreds of security officers, the leaders of France, West Germany, Italy, Japan, Canada, Great Britain and the European Community sometimes gave the impression that it was just a leisurely weekend away from home.

Shortly after they arrived Saturday, for instance, and presumably had unpacked their suitcases, some of them were out on Francis Street in the heart of the village, chatting and strolling around.

And this morning, Germany's Helmut Kohl arrived late at the House of Burgesses after an optional "Prayer for Peace" service at Bruton Parish Church. Kohl walked from one place to the other and on the way stopped in a silver shop to buy souvenirs.

Tonight, in the second of two dinners he's hosted, President Reagan welcomed everybody to the governor's palace. Margaret Thatcher had gone home; standing in for her was British Foreign Secretary Francis Pymm. It was the second night of food emphasizing America's southern cooking, with items such as "Cajun popcorn" (battered crayfish tails) and blackened redfish on the menu.

Earlier today, after their meeting in the same room where Patrick Henry delivered his famous "No taxation without representation" speech, everybody walked over to Raleigh Tavern for a seafood buffet luncheon. Even then the president and his visitors seemed taken by Williamsburg's tourist wares. At a photography stand in front of the tavern, there was a table set up with several handmade articles, including those by a silversmith, basket weaver, barrel maker and bookbinder.

Reagan, Canada's Pierre Trudeau and France's Franc,ois Mitterrand were intrigued with the barrel maker. Britain's Thatcher preferred the silver and inspected a coffee pot with a wooden handle. Even when it started to rain and she had to get out her umbrella, it didn't deter her from returning to the table for a closer look. The hand-bound books drew the attention of Italy's Amintore Fanfani and Japan's Yasuhiro Nakasone. In a lighthearted gesture, Fanfani picked up a hand-woven basket and put it on his own head for a minute, then on the head of German interpreter Heinz Weber.

White House Deputy Chief of Staff Michael Deaver told reporters that everybody was "getting along fine. It's been very harmonious." Other reports, however, indicated there were some tensions among the leaders over rising U.S. budget deficits and interest rates.

But according to Deaver, there hadn't been a single snafu. He then corrected himself to say that the Secret Service had dropped 14 Key lime pies Saturday night at Reagan's dinner in Carter's Grove on the banks of the James River. Deaver said they were able to put four of the pies back together again and laughed that he didn't know what the Secret Service was checking them for. "Maida Heatter's hardly subversive," he said of the famed dessert chef brought in by food expert Craig Claiborne for the summit.

The Saturday night dinner was the president's first of the weekend and he apparently enjoyed himself so much that it went overtime by more than half an hour.

Presiding in the small, candle-lit dining room where only the leaders sat with him, Reagan was very animated during the table talk, which focused on arms control and defense issues. While they talked and drank their after-dinner coffee, he took notes.

Reagan wanted everybody to be on a first-name basis and in the interest of brevity, perhaps, shortened Nakasone's first name to "Yasu." Reagan said it meant peace.

Throughout the discussion, Reagan continued to take notes and gestured to accent points he was making. Much of his conversation seemed to be directed toward Trudeau, who was sitting at one end of the table.

The dinner consisted of spicy American barbecue, North Carolina style, deep-fried catfish and hush puppies, Louisiana gumbo and cole slaw, cheese and chili cornbread, Key lime pie and black walnut ice cream. Everybody drank Chenin Blanc and Gamay Reserve.

Mitterrand praised the California wines, saying that when French grapes are diseased, growers send over California graftings to make them healthy again. But there was no report that he commented on the spicy food, although during summit preparations, his aides requested that he be served no sauces, and he is known to have a delicate stomach.

There was no champagne because there were no toasts, and there were no toasts because there were four dinners going on in four different parts of the mansion. The finance ministers, with their host Treasury Secretary Donald Regan, ate in the library. The foreign ministers, with host Secretary of State George Shultz, dined in the Plantation Room. The "sherpas"--the leaders' personal representatives--dined in the Drawing Room, presided over by W. Allen Wallis, undersecretary of economic affairs at the State Department.

Pecking order might have best been judged by the comfort of the stools offered to interpreters at the different dinner locations. For those translating for Fanfani, Kohl, Mitterrand and Nakasone, there were upholstered chairs. Those assisting the finance ministers got stools with red leather tops; those helping the foreign ministers sat on hard wooden stools. All the sherpas spoke English and needed no interpreters.

The china was Wedgwood green-and-white Chinese Tiger, and there was enough of that for everyone to eat from. Not so with the flatware. Reagan and the other leaders got sterling silver; the foreign ministers used stainless steel.

The leaders arrived individually in shiny black limousines bearing specially made Virginia license plates. Each carried the insignia of the summit and the country's designation such as "IT 1" for Italy's Fanfani and "JA 1" for Japan's Nakasone.

Reagan, ever the genial host, greeted each of his guests at Carter's Grove's front door, patting them all on the back (except Thatcher, whom he patted on the arm). Between arrivals, the president inspected the wood paneling in the foyer, which apparently so intrigued him that when he and Shultz left, after everyone else had, he stopped again to point it out.

Before dinner, everyone gathered in the back yard for cocktails and an official photo session. The White House invited the seven-piece Preservation Hall Jazz Band up from New Orleans and its members held forth while everybody milled around.

The president apologized for interrupting the band to make some official remarks. And when he was finished, to make up for it perhaps, sang the first line of "When the Saints Go Marching In" along with the band.

There were two other tents on the grounds, but they were beyond the view of the high-powered guests. Official staffers sat down at round tables covered with white cloths to have the same menu as the honorees. But in a considerably more informal setting of brown-checkered tablecloths, there was a buffet for the 200 or more support troops that included the security officers, helicopter pilots, motor pool drivers and representatives of the press. Among them was Mark Freeman, executive vice president of the Catfish Farmers of America, who provided the evening's catfish. "The catfish on the menu is the first meal in heaven," said Freeman.

However, food entrepreneur Claiborne, who brought chefs from around the country, said he hadn't had any reaction from the leaders on his menu selection. So he used his only available yardstick.

"I looked at the garbage can," he said, "and there was very little garbage."

That, apparently, was a good sign.