Massachusetts Republicans, already something of an endangered species in this liberal state, have compounded their problems with a bitter fight over the party chairmanship.

A coalition of moderate and conservative insurgents, led by the party's top vote-getters and former chairmen, has begun an effort to oust current chairman Gordon Nelson.

A Reagan Republican favoring Philip M. Crane (R-Ill.) for the 1980 presidential nomination, Nelson is either seen as a tireless party activist or as trying to spread his conservative doctrine to the detriment of the state party.

Nelson is mostly self-confident. "I come off the streets; I can handle any of this," he said.

Nelson proved his point Thursday night when the state committee gave him a 47-to-8 vote of confidence. He engineered the move after receiving a letter signed by 18 of 30 Republican state representatives urging him to resign.

"After the vote I held out the olive branch and told them if we don't stop this idiotic squabbling it's going to destroy our ability to get people elected," Nelson said today. "We still have people who want to refight the Rockefeller-Goldwater battle. That kind of attitude is a blueprint for disaster."

But Nelson's opponents, calling on "all Republicans unwilling to see the Republicans live in fear or die in disgrace," today vowed to elect their members to the state committee and defeat Nelson in the party elections March 4.

Leaders of the rebellion, like state representative Andrew Natsios, have not forgiven Nelson for abandoning the party's moderate candidate, then House minority leader Francis Hatch, in favor of conservative Democrat Edward J. King in last year's governor's race. King won.

"That kind of behavior is just outrageous," said Natsios, a conservative. "We don't want our party leaders involved in campaigns -- especially Democratic campaigns."

From his roost in the party's right wing, Nelson has since 1976 ruled the party that spawned Sen. Edward Brooke, Sen. Leverett Saltonstall, Ambassador Elliot Richardson and Transportation Secretary John Volpe.

Today, the party holds none of the executive offices at the state house and only 36 of 200 legislative seats and two of 10 positions on the state congressional caucus.

And with Sen. Edward M. Kennedy (D-Mass.) apparently gearing up for a presidential run, local Republicans are even more fearful that their already palsied hold on the electorate will slip away.

"Our disease is symptomatic of a national affliction, with only about 18 percent of the country calling themselves Republicans and just about 15 percent of the voters here," said state Senate Minority Leader John Parker. "The Massachusetts Republican Party is a death rattle."