Sen. Edward W. Brooke (R-Mass.), plagued by an acrimonious divorce during his victorious primary campaign, has "failed and refused" to honor his former wife's new financial settlement, according to a contempt-of-court petition filed by Remigia Brooke in Middlesex Probate Court yesterday.

The 14-page document charges that the embattled senator failed to pay his former wife $1,500 for her June alimony and $20,000 in legal and accounting fees.

It also alleges that the senator reneged on his agreement to transfer to his former wife the couple's homes in Newton. Mass., and the Caribbean island of St. Martin. The document further charges that Brooke did not pay the $16,350 mortgage due on the Newton house.

Brooke countered yesterday afternoon by "reluctantly" filing a contempt petition against his former wife for failing to sign the deed on the couple's Martha's Vineyard property. Under the new settlement. Brooke was to retain that property and his Watergate condominium in Washington.

Brooke's attorneys contend that Remigia Brooke's failure to sign the deed by Sept. 13 prevented the senator from obtaining a bank loan to meet his financial obligations to his former wife under the new divorce agreement.

The senator's only equity against which to obtain a mortgage was the Matha's Vineyard property, his attorney. Robert McGrath, said in a prepared statement. However, without Remigia Brooke's signature on the deed giving the senator clear title, the bank would not provide the loan, he said.

A check for $37.849, including funds for the alimony payment, had been issued by the bank but was later rescinded when deed remained unsigned, McGrath said.

A contempt hearing has been set for Dec. 28 in Middlesex Probate Court, the site of the senator's legal problems over the past two years.

Brooke agreed to the new, substantially larger settlement for his wife on June 14. The move followed several weeks of intensive negotiations between lawyers for both sides in an effort to avert a new divorce trial requested by Remigia Brooke at the height of the senator's campaign for a third term.

The agreement signed by Brooke "under the gun," ended the couple's 31 years of marriage. Meanwhile, the barrage of bad publicity, engineered by Brooke's daughter Remi Petit, 29, caused speculation early in the campaign that Brooke would drop out of the Senate race - resulting in much of his campaign contributions drying up through the summer.

Last Tuesday Brooke narrowly fended off a stiff primary challenge from conservative television and radio talk show host Avi Nelson. The senator faces two-term Democratic Rep. Paul E. Tsongas in the Nov. 7 election.