A probate court hearing to set a new divorce trail date for Sen. Edward W. Brooke (R-Mass.) and his wife, Remigia, was delayed for one day yesterday while attorneys began a new attempt to negotiate a property settlement.

Lawyers said only that talks were proceeding, but an aide to Sen. Brooke, Robert Waite, commented. "We're cautious about predicting success. Previous negotiations broke down June 27.

The failure of those bargaining sessons resulted in Mrs. Brooke's request then for a new divorce trail - a move that could severely reduce the senator's chances of election to a third term.

The divorce settlement originally was reached Dec. 15, 1977, and was to take effect June 15.

However, the case was reopened by Middlesex Probate Court Judge Lawrence T. Perera in light of Brooke's admitted "misstatements" about his financial condition.

The false testimony is the basis of an investigation by the Suffolk County district attorney here into the possibility of filing perjury charges against Brooke.

Rescheduled for later today were motions by Mrs. Brooke's attorney, George Ford, to freeze the senator's assests and retry the divorce case the last week of this month, only weeks before the Sept. 19 primary.

Also set to be heard today is a motion by the senator's attorney, Robert McGrath, asking for postponement of a new trial for as long as six months - well after the November election.

The main point of the negotiations, Waite said, continues to be who will pay the nearly $80,000 in transfer taxes on the couple's home on the Caribbean island of St. Martin. The home is estimated to be worth more than $300,000.

Meanwhile, the newly appointed special prosecutor investigating possible perjury charges against Sen. Brooke began assembling court documents from the divorce case.

Mrs. Brooke has charged that her husband lied to inflate his liabilities so that she would receive a smaller divorce settlement.

Former state Supreme Judicial Court justice Jacob J. Spiegel, chosen by Suffolk County District Attorney Garrett H. Byrne Wednesday to conduct the probe, said that before he can recommend prosecution for perjury he must determine whether Sen. Brooke intentionally deceived his wife when he made the "misstatements" about his finances.

"It would have to be a material representation which is false," Spiegel said. "But a man can make a false statement and it still cannot be perjury."

Spiegel, who served on the state's highest court from 1960 to 1972, was lauded by the legal community here in interviews yesterday.

Boston Attorney James St. Clair, President Nixon's lawyer at the close of the Watergate legal battles, called Spiegel "a very distinguished jurist" - a characterization echoed by bar association presidents and legal scholars here.

Spiegel, a Republican who worked as a speech writer and legislative aide to massachusetts Gov. Leverett Saltonstall and Sen. Henry Cabot Lodge, waved off suggestions of partisanship in the investigation, saying he has been a registered independent with no political affiliations since his appointment to the bench of Democratic Gov. Foster Furcolo.

Since retiring from the bench, Spiegel has been the presiding master appointed by Federal Judge W. Arthur Garrity in the controversial Boston school desegregation ease and a member of Sen. Edward Kennedy's judicial selection committee, which recommends appointments to the federal bench.

Spiegel, 76, denied he is too old to head this politically controversial probe. "I may be approaching anti-quity but still not senility," he said. "It's not a question of a man's age. "It's his integrity, character and ability."