A House Judiciary subcommittee yesterday gave an unexpectedly cool reception to controversial legislation that would allow prayer in public schools.

One subcommittee member after another joined representatives of major religious denominations in denouncing the prayer amendment, sponsored by Sen. Jesse A. Helms (R-N.C.).

Witnesses and legislators described the Helms amendment as being more pernicious than just a maneuver to skirt the Supreme Court's ban on state-prescribed prayers in public schools.

The hard line was set early in the day by John M. Harmon, an assistant attorney general, who said he was confident that the Justice Department would recommend a presidential veto of the measure if it were adopted by Congress.

Spokesmen for the National Council of Churches, the Baptist Joint Committee on Public Affairs and the Anti-Defamation League of B'nai B'rith agreed with Harmon that the Helms amendment would set dangerous constitutional precedents.

Their positions, calling for defeat of the legislation, were not unexpected.

More surprising was the general and bipartisan subcommittee reaction against the bill, which enjoys broad and vocal political support.

The long-simmering feud over prayer in school has been around Congress off and on for 20 years, this being the fifth major effort since 1963 to thwart the Supreme Court.

But this time around it has taken a different twist. The Helms amendment would remove all federal court jurisdiction any matters related to school prayer.

Under pressure from the evangelical Christian movement, the Senate last year passed the Helms amendment as part of an appellate court measure, then sent the political hot potato to the House.

The pressure intensified, and Rep. Robert W. Kastenmeier (D-Wis.), courts subcommittee chairman, was forced to call hearings when faced with the threat of losing control of the bill.

Hearings yesterday, and another session today, were scheduled after Rep. Philip M. Crane (R-Ill.) and the evangelicals got 178 signatures on a petition to force the bill to the floor. They need 218 for victory.

The evangelicals will have their opportunity to testify today, but they will encounter a subcommittee which, while apparently sympathetic to their aims, is leery of the constitutionality of their approach.

Rep. George E. Danielson (D-Calif.), among others, noted the intense emotion surrounding the school-prayer debate, but pointed out that the high court never has banned voluntary prayer in schools or public buildings.

"I am very troubled by the fact that by a majority vote of Congress we may, by something other than a constitutional amendment, be taking away constitutional rights," said Rep. Tom Railsback (R-Ill.).

Harmon said he was similarly troubled and that "the cleanest way to deal with the prayer issue is by a constitutional amendment."

"I want everyone here to understand we are not talking about a constitutional amendment." Danielson said. "At most we would create a law that could be held unconstitutional. The Helms amendment is only statutory." t