A possible technical error by the Carter administration has jeopardized the entire Selective Service registration system and could force nearly 9 million young men to register again at a cost of $18 million, attorneys said today.

Experts on federal law on both sides of the military draft registration issue said they were stunned by U.S. District Court Judge Terry J. Hatter Jr.'s ruling Monday that President Carter had violated a 30-day public notice rule when he put the registration system into effect in 1980.

They said it was rare for a judge at Hatter's level to throw out such a significant law on a technicality, but that his ruling had a chance of being upheld on appeal.

"It could go either way," said Harvard law school professor Gerald Frug. He said appeals courts were very unpredictable when faced with cases in which technical errors seem to invalidate major legislation.

The Justice Department said today it will appeal Hatter's ruling and continue investigating and prosecuting young men who allegedly have failed to register for the Selective Service.

Faced with perhaps 400,000 non-registrants, Selective Service officials say they have begun to send to the Justice Department names of young men who have not registered and who show up in Social Security records as being over 18.

Government spokesmen and attorneys said today they felt they had a strong case and argued that Carter's presidential proclamation setting up the system did not come under the 30-day rule.

One federal attorney, who asked not to be named, said even if Hatter's ruling held up, Congress could create a new registration program that would not require the 8,940,000 who have already signed up to register again.

But American Civil Liberties Union attorney Mark Rosenbaum, who argued against the Selective Service System before Hatter, said doubts about the validity of the old registration would create an administrative nightmare.

Hatter invalidated Carter's 1980 order, in the process of dismissing the government's case against former Yale philosophy student David Alan Wayte, 21, who had been indicted for failing to fill out a registration card.

Hatter agreed with defense attorneys that Carter had waited only 21 days from the time the new law was published in the Federal Register until he issued a proclamation putting the system into effect.

The Selective Service law, he said, required a 30-day wait, so that people would have time to become aware of its provisions.

Hatter also dismissed the case on three other grounds: that Wayte had been a victim of "selective prosecution" because only resisters who had written to the government announcing their defiance of the law or otherwise identifying themselves had been indicted; that the government had failed to give Wayte's attorneys internal government documents relevant to their case, and that the government had failed to produce presidential counselor Edwin Meese III to testify on how the government decided whom to prosecute first.

In his 37-page order and opinion, Hatter noted that the government had so far failed to prosecute thousands of non-registrants who had not identified themselves.

He quoted from a March 2 letter drafted by Justice Department official David J. Kline: "The chances that a quiet non-registrant will be prosecuted is probably about the same as the chances that he will be struck by lightning."

One activist in the anti-draft movement, ACLU legislative counsel David Landau, predicted the registration program would die if Hatter's ruling were upheld on appeal.

"Politically it's not going to happen," Landau said. "The Congress is not in a mood to get the system moving again."

Spokesmen for members of Congress who have been active in Selective Service legislation said the legislators had not yet been briefed on the impact of the Hatter decision and could not comment.

Hatter said: "The court recognizes the widespread effect that a decision granting defendant's motion to dismiss due to the illegal promulgation of the proclamation will have on this nation's Selective Service registration program.

"However, justice compels the court to grant defendant's motion . . . . What might appear a minor breach of the mandate of the Selective Service law takes on far greater significance in the context of criminal liability emanating from a defective component of the draft registration's statutory scheme."

Hatter, a former top aide to Los Angeles Mayor Tom Bradley, a former federal prosecutor and former state Superior Court judge, was appointed to the federal bench by Carter in 1979.

Asked about his 1980 proclamation at a news conference here today held to promote his new book, Carter said his legal advisers had told him at the time that the timing and form of the order were legal.

Several attorneys speculated today that the 9th U.S. Circuit Court of Appeals, which will hear the case next, might use the narrower grounds to dismiss the charges against Wayte, while leaving the Selective Service registration law intact.

Assistant U.S. Attorney Richard R. Romero, arguing the government's case, had told Hatter that Carter's proclamation did not come under the 30-day rule because the law referred only to "regulations" and not "proclamations."

He also said Carter had the right to waive the 30-day rule in cases involving harm to national defense and had effectively done so when, in the July 2, 1980, proclamation, he said he was "deeply concerned about the unwarranted and vicious invasion of Afghanistan by the Soviet Union."

Romero also argued that the error was harmless, done in good faith and should not invalidate an entire law.

So far 13 men have been indicted for failing to register and four have been convicted.

In two convictions, Carter's proclamation was not an issue but in two other convictions federal judges in Ohio and Iowa failed to accept the view that Carter had erred.