"That man is crazy," said a bewildered passer-by, pointing to the graying, barefoot man in a green T-shirt and shorts who was urging his guests to ride the elephant on the sidewalk in front of his house.

That explanation may be as good as any. Philanthropist Stewart R. Mott gave a day-long party yesterday with his colleague Anne Zill, ostensibly to celebrate their 10-year Washington work association. Mott, who is famous for throwing parties for various liberal causes, said, "Most people suspect that we have some secret purpose. But maybe once in 10 years it's okay to have a party just for fun."

Mott has thrown parties in cathedrals and at one-ring circuses, but this one was at his palatial white house on Capitol Hill's Maryland Avenue. Upon arrival, all guests were instructed to pay a penny poll tax and don a gold sash with the message: "Honi soit qui mal y pense," which was roughly translated by a Mott helper as "Curses on those who think badly of us." Guests were also informed that they were subject to arrest at any moment for "failure to vote, suspicious behavior and beliefs, and failure to enjoy." As one man was taken to the "jail" (behind the bar, of course), he commented, "Gee, I've never been to jail here in Washington. I frequent the South."

Zill, who is Mott's Washington funding representative ("I give away his money"), said that the party was to celebrate "a few good members of Congress and some people outside government who probably should be running the government, not because they are famous, but because they say good things." Guests who said "good things" came from such steadfastly liberal groups as Children's Fund, Catholics for Free Choice and the Sherwood Forest Foundation. Mott, who in the past has supported an array of candidates from Nelson Rockefeller to John Anderson to Marion Barry, said that he is supporting Sen. Alan Cranston (D-Calif.) for president this time around.

As guests wandered in and out of the house, they could view on Mott's walls: a display detailing the dangers of a nuclear arms race, a map of El Salvador, a population growth chart. Guests "voted" for the best and worst in Congress and for their presidential preference. Voter qualifications: "Are you male? Are you white? Do you own property?" and "Did you brush your teeth before breakfast this morning?"

Many of the guests congregated in the garden and in the front yard, where a rather large pile of elephant dung elicited a comment about Republican political views. Guests had been instructed to arrive in shifts, and children abounded during the first time block, from noon to 4, splashing in the birdbath.

In addition to the elephant, there were two donkeys tied to trees, presumably for political balance. The guests contentedly grazed alongside the animals, sampling stuffed grape leaves, corned beef, spring rolls and the ubiquitous wheel of brie.

An assortment of teen-agers in green T-shirts reading "ABZ-SRM-122-10" (Anne B. Zill, Stewart R. Mott, 122 Maryland Ave., 10 years) helped with the voting and registration of guests at the door. The entertainment changed throughout the party, from a belly dancer, with whom Mott danced enthusiastically, to a magician to a palm reader.

Among the guests were philanthropist Phil Stern, American Enterprise Institute fellow and author Ben Wattenberg, national security expert Morton Halperin and former Texas gubernatorial candidate Cissy Farenthold.

Arriving for the party's late shift were Reps. Lindy Boggs (D-La.) and Claudine Schneider (R-R.I.) and D.C. City Council Chairman David Clarke, who was "arrested" for failure to vote.

The towering Clarke's rejoinder: "Don't you know not to arrest somebody bigger than you?"

Earlier, while getting ready to pay his poll tax, presidential candidate Anderson announced, "I'm paying one penny and not a dollar, like some of these other big spenders."