Aides to President Carter and Sen. Edward Kennedy met privately yesterday to work out peaceful ground rules for their upcoming party convention contest.

Meanwhile, Democrats prominent in fund-raising efforts in campaigns past were conferring privately in a belated effort to prevent the party from nominating either Carter or Kennedy -- hoping to swing the convention to some other well-known figure such as Vice President Mondale, Secretary of State Edmund S. Muskie or Sen. Henry Jackson (D-Wash.).

The Carter and Kennedy representatives met for 2 1/2 hours yesterday afternoon in the neutral territory of a suite at the Mayflower. Afterward both sides said that they had made progress on procedural issues, and that there had been no attempt to forestall the main event, the planned challenge by the Kennedy forces to the convention rules. That is seen as a prelude to the vote on the presidential nomination itself.

"We talked about scheduling and procedural matters," said Richard Moe, vice president Mondale's chief of staff, who was attending the session as a Carter representative. "We discovered that, in a sense, both sides want the same thing -- a good convention. We both will debate our issues at the convention. . . . We assume there will be a rules fight and that the senator will continue his effort to win the nomination. And we assume that President Carter will win on the rules and win the nomination."

A spokesman for Kennedy said the meeting laid the groundwork for what both sides hope will be orderly procedures for the convention next month in New York City. He added that the question of Kennedy withdrawing from active candidacy was not raised, although Carter now has almost 300 more delegates pledged to him than he needs to win renomination.

"The senator is going to continue on to the convention as a candidate," said Richard Drayne, the Kennedy spokesman. ". . . We will continue our efforts on the rules and platform questions."

Representing the Carter side were chief campaign strategist Hamilton Jordon, delegate counter Thomas Donilon, and Moe. Representing Kennedy were Paul Kirk, Carl Wagner, Jack English and Harold Ickes.

The Carter and Kennedy advisers agreed to meet again next week to continue their talks, Moe said.

Meanwhile, in private meetings at the nearby Madison Hotel and in a series of long-distance telephone conversations, Democrats disenchanted with Carter and Kennedy were making their long-shot effort to turn the convention to some third choice. They are trying to put together a group they are tentatively calling the Committee for an Open Convention. They say they do not know if they have enough strength -- or enough time -- even to form the committee, let alone accomplish their goal.

"It's a longshot deal," said S. Harrison (Sonny) Dogole, a Philadelphia business executive who is one of the prime movers behind the late starting effort. "But Jimmy Carter is almost unelectable now, and it will be worse yet for him after Billygate." Dogole was referring to recent revelations of Billy Carter's financial ties to Libyans and his role in talks between the White House and a Libyan official.

Dogole, who has actively supported Sen. Jackson in the past, added that he was no more enchanted with Kennedy than with Carter. "Kennedy is unelectable, too," he said. "So sure it's late. It couldn't be later. What we need at the convention is a minor revolution."

Dogole said that "within the next four or five days" they will decide whether they have sufficient support and time to formally create a political action committee. Their plan is to have Washington attorney Edward Bennett Williams serve as the committee chairman.

"I was asked to head such a committee" said Williams, who is president of the Washington Redskins football team and owner of the Baltimore Orioles baseball team. "But to my knowledge no such committee has been formed."

Much of the exploratory work for the committee has been conducted by Arnold M. Picker, former executive of United Artists and financial chairman of Muskie's 1972 presidential campaign.

From his room at the Madison, Picker has been seeking to enlist prominent elected officials, which he concedes is a requisite for the formation of his committee. But as of yesterday he said he had none to announce. He and his colleagues hope to convince delegates at the convention to vote down a proposed rule that would bind delegates to vote for the candidate they were elected to support in their state primaries or caucuses. Only then could Carter be denied a first-ballot victory -- and only then could the delegates perhaps swing to a third choice.