Massachusetts' senior senator, Edward M. Kennedy, got the home-state boost he badly needed tonight, beating President Carter by better than 2 to 1 in the delegate-rich Democratic presidential primary here.

With 86 percent of the precincts reporting, Kennedy had 66 percent of the vote to Carter's 29 percent. California Gov. Edmund G. (Jerry) Brown Jr. ran a distant third with about 3 percent.

As Kennedy was scoring his first victory of the 1980 campaign, Carter rolled up a 3-to-1 victory in the nonbinding "beauty contest" primary across the border in Vermont.

Kennedy's win in Massachusetts was greater than a statewide poll last week had predicted, but it was not clear whether he could translate a home-state triumph into that cherished commodity known as momentum.

Nationally, tonight's results will probably leave his campaign as it was before -- way behind but not hopeless.

Kennedy, standing arm-in-arm with his wife, Joan, and flanked by a dozen relatives, claimed victory before a joyous crowd of supporters here. "If our candidacy means anything -- and after tonight it means something -- it is that the American people understand that the No. 1 issue is . . . the economy . . . that the American people will not tolerate inflation of 20 percent," he said.

Carter, in a statement issued in Washington, said: "I deeply appreciate the expression of strong support from the voters of Vermont and wish to congratulate Sen. Kennedy and his campaign organization on their victory in Massachusetts."

Carter thanked his campaign volunteers, saying that "while I am remaining in Washington to manage the nation's affairs, they are doubly important."

Kennedy will score a big gain in convention delegates when the Massachusetts results are complete.

The state will send 111 delegates to the August Democratic National Convention -- more than the total of delegates from Iowa, Maine and New Hampshire, where Carter defeated Kennedy earlier.

But with Kennedy's big win here, it was likely that he would wake up Wednesday leading the president in convention delegates.

Associated Press projected a Kennedy capture of 77 Massachusetts delegates to Carter's 34, which would give the challenger a 100-to-84 advantage overall.

Kennedy's lead is not likely to last long, however. Primaries next week in Florida, Georgia and Alabama will allocate 208 delegates and Kennedy has effectively conceded victory to Carter in the president's native South.

Kennedy reiterated tonight that he would go all-out in primaries in Illinois two weeks from today, and New York, three weeks away. Polls in both states show him trailing.

Last night, both Carter and his national campaign chairman, Robert S. Strauss, stressed the number of primaries and caucuses occuring in the next two weeks. More than 400 convention delegates will be at stake, Strauss said, predicting that Carter "will do very well" and win more than a majority. He specifically predicted a victory in Illinois March 18.

As for tonight's results, Strauss said he was "pleased with respect to Vermont, which went better than we expected, and we're satisfied that Massachusetts is going about how we expected."

Polls by ABC and NBC News of voters leaving Massachusetts polling places indicated that Kennedy beat Carter in nearly every age group, income class and ethnic or religious category. The only Democrats who seemed to have supported Carter consistently were those who described themselves as "conservative."

ABC reported that, among Democrats leaving the polls, 70 percent who said they were Catholics said they voted for Kennedy and 24 percent said they went for Carter. Among Jews, the figures were 69 percent for Kennedy and 26 percent for Carter. Those who said they were atheists went for Kennedy 66 to 30 percent, and Protestants split 50 to 47 percent in favor of Kennedy.

The Democrats rated their senior senator higher than Carter in nearly every aspect of job performance, but the voters who said that "honesty" was an important factor in their vote gave Carter a small edge.

Kennedy based his campaign here partly on appeals to state loyalty and partly on sharp attacks on Carter's economic policies and the soaring inflation and interest figures the president has been unable to combat.

Kennedy told Democrats here that Carter was a loser and his renomination would result in a Republican victory in November.

Carter's extensive media campaign dealt mainly with character questions. The ads suggested that Carter is honest and Kennedy cannot be trusted to keep his word.

In the opening days of the Kennedy campaign, when victory in Massachusetts, and most other places, seemed almost a certainty, the senator's campaign staff joked about Massachusetts as "the first real test." "After all," the Kennedy people would laugh, "that's where they know us best."

But the humor went out of the situation in the last two weeks, when Kennedy's reverses elsewhere, and stepped-up Carter campaigning in the Bay State, spawned rumors that the race was tighter than anyone had expected. b

Having failed to land a hoped-for knockout punch in New Hampshire last week, the Carter camp mounted a strong final-fortnight drive here in the belief that a win, a tie or a close second by the president might end Kennedy's campaign.

Carter spend about $90,000 on media advertising in Massachusetts, according to his Boston headquarters -- 50 percent more than Kennedy did. The president worked the telephone and dispatched his platoon of stand-ins, including wife Rosalynn and "vacationing" White House staff members, in the effort to embarrass Kennedy at home.

Carter was endorsed by a number of Democratic leaders, including Gov. Edward J. King and the leadership of both houses of the state legislature. This reflected partly the local politician's discomfort with Kennedy's liberalism and partly a resentment that is traditional in the home state of the party's most-successful family.

"The rest of us had to go to all the Columbus Day parades and every pancake breakfast at St. Timothy's," a Democratic old-timer said the other day. "But not the Kennedys. If they did show up, it was a big deal, but if they missed, nobody blamed them."

The president was expected to run well in Boston's black and working-class white neighborhoods.

Many white voters have not forgiven Kennedy for his support of school busing; some black voters resent his campaigning for fellow Democrat Paul Tsongas in 1978, when Tsongas unseated Republican Edward W. Brooke, the only black in the Senate.

Kennedy, for his part, was relying on the relative liberalism of Massachusetts Democrats. His skepticism about Carter's recent tough, militaristic gestures toward the Soviet Union and his attacks on the president's decontrol of oil prices seemed likely to score with many party members here.

Brown established a national campaign headquarters in Boston three months ago and had planned an agressive campaign aimed at the college population. But after his poor showing in New Hampshire (10 percent of the vote), he shut down his operation here.