A Montgomery County judge ruled yesterday that former Securities and Exchange Commission official John M. Fedders is not entitled to a financial share of his former wife Charlotte's book about the domestic violence that led to their widely publicized divorce.

Also, the judge rejected the opinion of a court-appointed domestic relations master who held Charlotte Fedders equally to blame for the breakup of the marriage.

Circuit Court Judge James S. McAuliffe was acting on Charlotte Fedders' appeal of the domestic relations master's October decision awarding John Fedders a 25 percent share of the proceeds of her book, "Shattered Dreams." In justifying the award, the master, John S. McInerney, had said Charlotte Fedders shared blame for her husband's violent outbursts because she withheld emotional support during his spells of depression.

McAuliffe said yesterday that regardless of what caused John Fedders to strike his wife, "it doesn't change the fact that he did it."

"The legal fault lies with John Fedders, clearly and uneqivocally," the judge said, adding: "What he did was as classic a case of cruelty of treatment as one is likely to find."

But McAuliffe overruled McInerney's decision on the book proceeds for a different reason. Because no exact monetary value can be placed on the book, he said, it cannot be considered a marital asset.

Charlotte Fedders said she was "overwhelmed" with joy at McAuliffe's rulings.

"Finally, a very fair and very just conclusion," she said. "The fact that he decided the fault question once and for all, that's vindication for me and for many, many battered women."

John Fedders, who resigned as enforcement director of the SEC in 1985 amid the widely publicized abuse accusations, was not in the courtroom yesterday. His lawyer, Hal Witt, said his client would have no comment.

McAuliffe's ruling came several hours after Witt made a surprise announcement in court that his client has decided to give up his claim to a financial share of the book.

Witt told the judge that John Fedders believes that distribution of the book has caused psychological harm to the family's five sons. John Fedders "thought long and deep" and decided that, as a matter of conscience, he did not want to profit from a book that harmed his children, Witt said.

However, he said that John Fedders continues to believe that

McInerney's decision was correct "as a matter of law and equity."

Charlotte Fedders and her lawyer, Bryan Renehan, scoffed at John Fedders' renunciation of his claim. Renehan said there was little chance of the book award's being upheld by McAuliffe and that John Fedders "was trying to make himself look like a good guy" in the judge's eyes, hoping for favorable rulings on other issues that were decided yesterday.

As for those other issues, McAuliffe reduced John Fedders' monthly alimony payments but generally ruled in Charlotte Fedders' favor on the questions of visitation rights with the children and the division of money from the sale of the couple's house in Potomac.

After filing for divorce in 1983, Charlotte Fedders was awarded $750 a month in alimony. McInerney reduced the figure to $500 after John Fedders testified that his once-bright legal career had been ruined and his income had been sharply curtailed. Yesterday, McAuliffe further reduced the alimony payments to $400 a month.

When the Fedders' $450,000 Potomac house is sold, however, Charlotte Fedders will receive half the proceeds plus $50,000, McAuliffe ruled. McInerney had ordered the proceeds split evenly.

Domestic relations masters, who are lawyers, decide on divisons of assets in divorce cases. By law, McInerney's October decision was held in abeyance while Charlotte Fedders appealed it to McAuliffe.