A Montgomery Circuit Court judge agreed yesterday to bar the public from the divorce trial of former Securities and Exchange Commission enforcement director John M. Fedders, only to reverse himself hours later after legal counsel for two newspapers protested.

Circuit Court Judge James S. McAuliffe, in an action he called unprecedented in 30 years of trial experience, ordered the courtroom cleared of spectators and news reporters yesterday morning after a lawyer for Fedders argued that media coverage of the case might embarrass the Fedders' children and cause them to be harassed at school.

The Fedders have five children, aged 3 to 15.

"As of this point I'm following my nose," McAuliffe said after closing the proceeding.

"I feel in my heart I'm doing the right thing."

By late afternoon, however, after a special hearing in which a lawyer for The Washington Post and the Journal Newspapers argued in favor of an open trial, McAuliffe reversed himself.

"How can you close Fedders versus Fedders and not close any other case where the parents are at war and their children are going to be affected at school the next day?" the judge asked. "In the final analysis I can't make a distinction . . . . "

The legal skirmish dominated the first of day of the trial, in which Fedders' wife of 18 years, Charlotte Fedders, is seeking a divorce and custody of their children.

At the start of the hearing on access late yesterday afternoon, Charlotte Fedders turned away from her husband the only time he attempted to speak to her. She left the courtroom sobbing during testimony by a psychologist about the potential effects the case would have on their children.

"The children had some difficulties at school that I feel we worked very hard to stabilize. We shouldn't do anything to jeopardize them," testified Mary R. Donahue, the psychologist.

Lawyer Michael S. Horne, who represented the newspapers, argued, "It may be regrettable that children are involved here, and maybe uncomfortable, but that's the price we pay for living in a society with a First Amendment."

On its face, the Fedders marriage had an almost storybook quality. John and Charlotte Fedders were the children of Catholic families, and they had met in college. But their private lives were stormy from the outset, according to court records.

In pretrial testimony, Charlotte Fedders, 41, described herself as the "classic abused wife" who suffered in private while her husband, 43, a former Marquette University basketball player, rose in prominence -- first as a lawyer with a high-powered Washington firm and then as one of President Reagan's top law enforcement officers.

After hearing the president promise in his 1984 State of the Union Address to "intensify our drive . . . against horrible crimes like sex abuse and family violence," Charlotte Fedders wrote a letter to him about her situation.

In the letter, she stated that her husband had beaten her periodically during 16 years of the marriage, causing her at times to receive a broken ear drum, wrenched neck, black eyes and bruises.

Once, she wrote, he beat her around the abdomen while she was pregnant.

She never mailed the letter, but several months later it was passed to White House Counsel Fred Fielding by one of her sisters. The White House apparently never acted on the matter.

Fedders resigned his position Feb. 26, three weeks after reports about their relationship first appeared in the news media.

After Charlotte Fedders filed for divorce, McAuliffe twice granted delays when John Fedders said he wanted a reconciliation with his wife.