The presidential candidates tried again yesterday to devise an acceptable format for campaign debates, but President Carter again refused to participate on any terms other than his own -- setting the stage for a first debate that would feature Ronald Reagan, John B. Anderson and an empty chair.

In a 2 1/2-hour negotiating session here, representatives of Republican Reagan, independent Anderson and the debate sponsor, the League of Women Voters, proposed new formats to persuade Carter to face his two rivals.

But Carter's representatives rejected the new proposals, standing firm on their demand that Anderson be excluded from the first Carter-Reagan debate.

Further negotiations seem likely, and officials in the three campaigns said a resolution of the stalemate is still possible. As of now, though, it appears that the 1980 presidential debates will open Sept. 21 on a Baltimore stage with Anderson and Reagan -- and the chair.

Ruth Hinerfeld, chairman of the league's education fund, said yesterday that there probably will be an empty chair in Carter's place if the president does not change his mind about the Baltimore debate.

Hinerfeld, the debate organizer, had said Tuesday that there would be no empty chair. Asked again yesterday, she said, in tones that reflected her evident frustration with Carter's position, that it is standard league practice to provide chairs for candidates who skip debates "for their own purposes."

"I believe we will do so in Baltimore," she said.

Carter told reporters at the White House yesterday that he will not show up for the first debate if Anderson is present. "And if the other two . . . decide to debate as a Republican duo, to debate each other, that's perfectly all right with me," the president said.

In his effort to get Carter to debate with him and Anderson, Reagan is sounding more and more like Sen. Edward M. Kennedy (D-Mass.), whom Carter persistently refused to debate during the primary campaigns.

Leaving his hotel in Cleveland during a campaign stop today, Reagan was asked how he would characterize the president's refusal to debate.

"I think Teddy Kennedy was right, " Reagan replied. "You can't blast him out of the Rose Garden."

Carter and Reagan accepted the league's initial invitation to a series of debates this fall, but then Carter threatened to pull out unless he got to debate Reagan one-on-one in the first debate of the series. The president's managers say they fear that Anderson's inclusion in the first debate -- which historically draws the biggest audience -- would enhance the independent's campaign at Carter's expense.

Reagan, agreeing that Anderson takes votes mainly from Carter, sided with Anderson in demanding that he be included.

Tuesday the league reviewed recent poll data and determined that Anderson has "significant" support; the independent was thus invited to all three league debates. Carter promptly announced that he would not take part.

Yesterday morning, representatives of all three candidates met with Hinerfeld at the league's headquarters here to negotiate further. Hinerfeld said the league still preferred three debates featuring all three candidates, but said she would consider other formats to induce Carter to participate.

Reagan's camp, the participants said later, then proposed a "round-robin" set of three debates in which each candidate would face the others one-to-one: Carter-Reagan, Cater-Anderson, Reagan-Anderson. The order of these debates would be determined by lot.

Anderson agreed to this plan, but Carter's people vetoed it because it left a chance that Carter would have to debate Anderson before he got to Reagan.

Then the negotiators came up with a schedule that called for the initial three-man debate, to be followed by a Carter-Reagan debate under league auspices. The Reagan and Anderson camps agreed to consider this idea, but Carter's people vetoed it.

Carter's representatives said again that he would debate only if his first appearance is alone with Reagan. They said that, after the Reagan-Anderson debate in Baltimore, the league should sponsor a Carter-Reagan debate -- and only then would the president face Anderson.

Hinerfeld, noting that this was no different from the position Carter has held for the past three weeks, said it was rejected because, unlike the "round-robin" plan, it would not give Anderson an equal chance to meet Carter early in the series.

Negotiations broke up at that point. All sides later expressed an interest in trying again, though, and officials at the three campaigns all said there was still hope for a solution that would get all three men involved in the debates.

They said public reaction to events of the last two days might spur some or all of the participants to soften their positions.