Sen. Edward W. Brooke's (R-Mass.) wife, Remigia, yesterday requested a new divorce trial after around-the-clock negotiations between lawyers for both sides in the bitter dispute broke down just minutes before the court-imposed deadline for her decision.

Brooke - under the gun during the talks because of increased political problems stemming from the renewed divorce proceedings - turned down a last-minute proposal that would have given his wife a larger settlement than he agreed to last year, according to the senator's press secretary, Robert Waite.

"It was impossible for him to accept it financially," Waite said.

Remigia Brooke's attorney, George Ford, said the bottleneck resulted from "financial matters thaat couldn't be accommodated. The problems were all financial."

"This has been the most incredible negotiations I have ever been involved in," said the senator's attorney, Robert McGrath. "I don't think there was any good faith on the part of Mrs. Brooke or her daughters."

Ford, who said, "I think there were good-faith efforts on both sides," filed six motions to be heard in Middlesex Probate Court here tomorrow. One motion requests a freeze of the senator's assets. Another moves to retry the divorce case, starting the last week in July, a little more than a month before the Sept. 19 primary. The divorce was originally settled Dec. 15, 1977, with the settlement to take effect June 15

The new trial - which, under normal circumstances in a contested divorce, might take as long as 13 weeks to come to court, according to register probate Paul J. Cavanaugh - is potentially disastrous to the embattled senator's chances of reelection to a third term in November.

The senator's attorney, McGrath, filed an appeal Monday of the judge's decision granting Mrs. Brooke a new trial - as a hedge against no settlement.

He asked the judge to stay the new trial order until after that appeal is heard, a move he said could take six months.

McGrath said one of the "sticking points" in the talks, which broke down around 1:15 p.m. yesterday, included a requests that Brooke pay capital gains taxes that might be incurred in the transfer of the couple's house in Newton, Mass., to Mrs. Brooke, and also pay capital gains taxes that might be incurred in the transfer of the couple's house in Martha's Vinyard, Mass., to the senator.

Another, he said, was a requests that Brooke pay his wife's attorney's fees over the nearly 2-year divorce case, no estimate of the extent of the fees was available, but Mrs. Brooke's attorneys said that expenses in connection with two day days' worth of preliminary hearings earlier this month cost Mrs. Brooke $7,296.

Under the original settlement, Mrs. Brooke was to receive $18,000 a year in almony, the Newton house and one on the West Indies Island of St. Martin.

Since the divorce was reopened because of Brooke's admitted misstatements on his finances in a court deposition during his divorce, apparently inflating his liabities, Mrs. Brooke has also asked for payment of an estimated $80,000 in transfer taxes on the sale of the more than $300,000 St. Martin home.

However, a source who said he was authorized to speak for the family said Brooke had rejected an offer that his wife pay up to $50,000 on the island house in exchange for the senator's paying all other transfer taxes, no matter how high they went.

In the new settlement proposal, Mrs. Brooke also wanted all her medical and dental bills paid for by the senator during her lifetime. In the earlier agreement, he was required to pay only medical insurance premiums.

Also, in the new settlement proposal, Mrs. Brooke requested that she not only receive the Newton house, but that the senator pay off the outstanding mortgage on it.

Brooke refused to comment on the new developments, which will drag out the end of his 31-year marriage. However, McGrath said, "He's very sad. He wanted to resolve it get this thing settled. He really wanted an agreement; he really wanted to do the right thing for Remigia and the children."

Word of Mrs. Brooke's decision came from the judge yesterday afternoon in a one-line statement to his court clerk as jackhammers blasted outside the steamy courtroom. "Do I understand that in the Brooke case the plaintiff has requested a new trial?" he asked. None of the parties in the divorce was in the court.