Negotiators for President Carter and Sen. Edward M. Kennedy yesterday announced a treaty governing their clash next week at the Democratic National Convention, including an understanding that there will be some public display of "reconciliation" between the two when the fight ends.

The agreement, reached during a five-hour meeting between top Carter and Kennedy aides in a downtown hotel yesterday, essentially established a schedule for clashes over the party rules and platform. It aims at preventing the presidential nominating convention from turning into a bloody show of intraparty warfare.

The agreement, announced in a joint statement by the two campaigns at the end of the meeting, also all but pledged the losing candidate -- at this point almost certain to be Kennedy -- publicly to support the winner.

"Whatever differences we may have, they pale in comparison to our common differences with the Republicans and their nominee," the statement said. "Whoever is on our ticket, we are determined to conclude our convention united behind our nominees. With so much at stake in this presidential election, the Democratic Party must prevail in November."

The statement added that the two sides are committed to conducting a convention "that permits reconciliation at its conclusion."

There was no specific commitment in the statement that the president and the Massachusetts senator, who have battled for months through the primaries and caucuses, will get together in a show of unity after the nomination is finally settled Wednesday night. But a Carter aide said there is now an assumption on both sides that "we will all get back together at the end of the convention."

Yesterday's agreement set early Monday night for the most critical test of strength at the convention -- the battle about the rule that binds delegates to vote for the presidential candidate they are pledged to support.

Kennedy is basing his last-ditch effort to wrest the nomination from Carter on the hope of winning this rules fight. He has been joined by a number of other Democrats who fear that the president faces almost certain defeat at the hands of Republican nominee Ronald Reagan in November and who are calling for rejection of the rule and for an "open" convention.

The Kennedy strategists had hoped to delay the rules fight until Tuesday, giving them an extra 24 hours to lobby delegates in the unpredictable atmosphere of a national political convention. But yesterday, they agreed to begin a one-hour debate about the rule at 6:30 p.m. Monday, assuring that a portion of the debate and roll-call vote will be seen on prime time television. p

The two sides also agreed that a two-hour debate on the economic planks of the party platform -- the issues on which Kennedy has based his challenge to the president -- will take place during the prime time television viewing hours Tuesday night. Carter aides said they will be "amazed" if Kennedy does not use this time slot to deliver a major address to the convention on the night before the roll call for the nomination.

In return, the Kennedy aides agreed to reduce substantially the number of other minority reports to the platform and to dispose of them Tuesday afternoon before there is much of a television viewing audience.

Carter aides were clearly cheered by the agreement, which they said should mean that most of the prime time television hours at the convention in New York City would be devoted to attacks on Reagan and the Republicans rather than the spectacle of a protracted Kennedy-Carter clash.

"This lays the basis for reconciliation and unity after the convention," an official said.

However, while the agreement refferred to the need for unity and reconciliation at the end, neither side can be sure at this point what the mood will be after three days of undoubtedly emotional battles over rules, platform and nomination.

On Capitol Hill yesterday, before the agreement was announced, Kennedy sidestepped a question on whether he will support the party's nominee.

"I have indicated I believe the Democratic Party has to stand for its tradition values and its traditional historical values and its traditional historical commitments to the senior citizens, the working people, the middle income, the young people of this nation," he said. "I have absolutely no hesitation supporting that kind of a nominee and that kind of a platform."

Columnists Rowland Evans and Robert Novak reported today that Sen. Abraham Ribicoff (D-Conn.) as agreed to lead the fight by the Carter forces Monday night to preserve the rule binding delegates.

The selection of Ribicoff, a leading liberal with longtime ties to the Kennedy family, appeared a shrewed choice by the Carter campaign. The Connecticut senator was subject to a nationally televised oral assault by the late mayor Richard J. Daley of Chicago at the 1968 Democratic convention, the bloody confrontation that set in motion a decade-long effort to "reform" the delegate selection rules.

Meanwhile yesterday, Kennedy kept up his quasi-campaign activities by meeting privately with two senior Democratic senators who are publicly backing his call for an "open convention" -- Majority Leader Robert C. Byrd (W.Va.) and Henry M. Jackson (Wash.). He told both they were on his list of vice presidential possibilities but did not offer the position to either, Kennedy said. He said he does not exclude the possibility of naming his choice of a running mate before the convention.

"I am absolutely convinced that we are going to have an open convention," Kennedy said, adding that to do otherwise would make "robots' 'out of the delegates.

However, while Kennedy was making this optimistic prediction, a spokesman for Rep. Michael Barnes (Md.), one of the instigators of the "open convention" movement, said that the Massachusetts senator's continued presence in the race is hurting the effort to persuade Carter delegates to abandon the president on the rules issue.