The Justice Department said yesterday that a controversial court-curbing bill passed by the Senate earlier this year--the most restrictive anti-busing bill ever to pass either house of Congress--is in its opinion constitutional.

At the same time, however, Attorney General William French Smith said he had questions about the constitutionality of a bill drafted by Sen. Jesse Helms (R-N.C.) to strip federal courts of jurisdiction in school prayer cases. But he said the Justice Department has the duty of defending such a law, if it is enacted.

The Helms prayer bill could become moot given President Reagan's announcement yesterday that he plans to submit a proposed constitutional amendment to Congress to allow voluntary prayer in public schools.

Civil rights and legal groups quickly attacked Smith's non-binding but long-awaited opinion on the anti-busing bill, which they say violates the Constitution simply because it limits the authority of the federal courts to interpret the Constitution.

But conservatives, particularly southerners, were pleased by the move, which came only two days after President Reagan had disappointed them by endorsing a compromise plan backed by liberals and moderates for extension of the 1965 Voting Rights Act.

Since the administration took office, the attorney general has been under continual pressure from conservative groups, including the Moral Majority, to take positions on dozens of bills to limit the jurisdiction of the federal courts in controversial social areas like abortion, school prayer, and busing.

The anti-busing bill, which was cosponsored by Helms and Sen. J. Bennett Johnston (D-La.), would allow the federal courts to retain jurisdiction in school desegregation cases, but would prevent those courts from ordering that students be bused more than five miles or 15 minutes from their homes for racial reasons.

The bill, which was attached as a rider to the Justice Department's spending authorization bill, also forbids the department to initiate suits seeking busing as a means of desegregation and allows the department to attempt to remove or reduce existing busing orders.

"Careful examination of these provisions indicates that they are constitutional," Smith said in a 16-page letter to Rep. Peter W. Rodino Jr. (D-N.J.), chairman of the House Judiciary Committee, where the bill is bottled up.

Smith made the point that the legislation restricts the remedy, not the court's jurisdiction. But the language in his letter raises the question of whether Congress has the right to limit remedies: "Congress cannot impose mandatory restrictions on federal courts in a given case where the restriction would prevent them from fully remedying the constitutional violation."

On school prayer, Smith said he had major concerns about limiting court jurisdiction: "History counsels against depriving the court of its general . . . jurisdiction over federal questions. Proposals of this kind have been advanced periodically, but have not been adopted since the Civil War."

However, Smith said that if the legislation is enacted, "It is settled practice that the Department of Justice must and will defend acts of Congress . . . ."

The attorney general's split position on the jurisdiction issues did not appease civil rights groups.

Joseph Rauh, general counsel to the Leadership Conference on Civil Rights, a coalition of 161 organizations, said, "Once more the administration has surrendered to Jesse Helms. But the American public believes in the federal courts as a defender of individual rights and will force the administration to reverse itself here just as it did on the voting rights bill . . . ."

David Brink, president of the American Bar Association, warned recently that the court-stripping bills "endanger our system of government . . . a future Congress could wipe out federal jurisdiction and remedies in all constitutional cases . . . . Once the door is opened, the rights of free speech, free press, free assembly and free exercise of religion could be the next to be abolished or curtailed."

Laurence Tribe, a professor of constitutional law at Harvard Law School, said, "They're saying they won't prevent the court from listening to cases, they're just preventing the court from doing what it's supposed to do . . . . No matter how necessary busing might be, the court's arms would be cut off."