A long and bitter battle with Carter-Kennedy overtones ended yesterday with a decision to allow Massachusetts Democrats to hold the second primary of the 1980 presidential season.

In a move that could benefit homestater Sen. Edward M. Kennedy's presidential hopes, the party's Compliance Review Commission agreed to let the Bay State hold its primary March 4, one week after New Hampshire's first-in-the-nation primary.

The body also voted to allow Democrats in Maine, supposedly another Kennedy stronghold, to hold their party caucuses Feb. 10. Both are outside the March 11 to June 11 "window" created by the party in an attempt to shorten the primary season.

The move to exempt Masachusetts from party rules came after Carter supporters had complained bitterly about the behavior of Massachusetts Democratic chairman Chester Atkins, a Kennedy supporter who once complained the rules amounted to "an incumbent president relief act."

Ostensbily, the exemption was made because other party leaders in the state had made an unsuccessful, good faith effort to change the primary date, set by state law.

But it also represented an admission by Carter forces that they were boxed in on the issue. If the commission had refused to grant the exemption, the primary would still be held on March 4, but convention delegates would be selected by party caucuses at a later date.

Carter supporters fear they would do even worse in a caucus than in a primary. There also has been a rethinking in the Carter camp of the importance of the primary.

"I just don't think a defeat by Kennedy in a Massachusetts primary is a significant political event," said Jack Walsh, political director of the Carter campaign. "Losing to Kennedy in Massachusetts isn't earth-shaking."

The Massachusetts vote was a well-orchestrated political drama, its cast of characters intriguing. Atkins, the Massachusetts chairman, didn't show up. Instead, he sent James Roosevelt Jr., a grandson of Franklin d. D. Roosevelt, to represent his case.

The Carter case on the supposedly impartial commission was carried by Dwight Pettit, Carter's 1976 campaign co-chairman in Maryland. Ben Alexander, his 1976 New Mexico chairman, and William Armstrong, his 1976 Texas chairman.

Just when the exemption measure appeared doomed, Massachusetts House Speaker Thomas McGhee, a gruff-talking old Irish pol who has been courted feverishly by the Carter White House, gave an impassioned speech. "For the best interest of the Democratic Party, grant a waiver," he insisted.

The exemption passed unanimously.

Massachusetts is the fourth state to be exempted from party rules. The others are New Hampshire, which will hold a Feb. 26 primary, Minnesota, which will hold party caucus the same date and Iowa, which has precinct caucuses on Jan. 21.

In another move yesterday, California Gov. Edmund G. (Jerry) Brown Jr. said he intends to run in the Iowa caucuses, a move calculated to make him eligible to participate in a debate among presidential hopefuls in that state. Brown previously had been excluded from the debate because he had said he wouldn't campaign in Iowa.