President Reagan took on the nation's major teachers' organizations here today, saying public schools are failing and that one way to improve them without more federal funds is to start paying teachers according to merit rather than seniority.

That is anathema to most teachers' groups, who say there is no accurate way to measure teacher quality and that such traditional signs as pupil test scores are misleading.

In a commencement speech at Seton Hall University, Reagan criticized the schools, particularly the high schools, for "not doing the job they should."

"Again and again, when compared to students in other industrialized nations, American students place badly," he said.

Implicitly defending his education budget cuts, Reagan went on to say: "There was a time not too long ago when the solution to this problem would have been summed up by most politicians in one big five-letter word: money."

"They tried that approach, and it failed," he continued. "We spend more money per child for education than any other country in the world--we just aren't getting our money's worth."

"One of the best ways to do this," he said, "and unfortunately it is opposed by some of the heaviest hitters in the national education lobby--is by rewarding excellence. Teachers should be paid and promoted on the basis of their merit . . . . Hard-earned tax dollars should encourage the best. They have no business rewarding incompetence and mediocrity."

Reagan has little to lose politically in taking on the nation's large teachers' organizations. The National Education Association and the American Federation of Teachers are both active in Democratic politics.

Emphasizing another of his differences with public school groups, Reagan also touted his proposals for tuition tax credits and education vouchers, saying "We can . . . encourage excellence by encouraging parental choice."

In his speech at the 127-year-old Catholic university, where he was awarded an honorary doctor of laws degree for his opposition to abortion, Reagan also repeated past endorsements of prayer in public schools.

"I know this idea is not too popular in some sophisticated circles," he said, but "I can't help but believe that voluntary prayer deserves a place in our nation's classrooms."

Several times in recent weeks Reagan has spoken out on education, which some think could be an issue in next year's campaign. A national commission set up by Education Secretary T.H. Bell reported recently that U.S. schools are failing in their task, and Reagan endorsed that report.

Reagan pointed to several signs of weakness in the educational system yesterday, some of which were cited by Bell's commission.

"Since 1963, Scholastic Aptitude Test scores have demonstrated a virtually unbroken decline," Reagan said. "Thirty-five of our states require only one year of math for a high school diploma and 36 require only one year of science . . . . It has been estimated that half of our country's gifted young people are not performing up to their full potential."

He stuck to his anti-federal theme on education: "The road to better education . . . cannot be paved with more and more recycled tax dollars collected, redistributed and overregulated by Washington bureaucrats."

But Reagan also said that "there is much that the federal government can do to help set a national agenda for excellence in education," and "in the weeks ahead, I will have more to say on this subject."

"Perhaps the biggest irony about the problems facing American education today is the fact that we already know what makes for good schools," Reagan said, citing "leadership from superintendents and principals, dedication from well-trained teachers, discipline, homework . . . ."

He added: "All of these things can be improved without increased federal funding and interference, and with only modest increases in local and state support."

In addition to asserting himself on education, White House aides said one of Reagan's purposes in coming here today was to reach out again for the ethnic blue-collar vote and particularly the Catholic vote, which recent polls have shown Reagan losing since the 1980 election.

The president's tough remarks about the nation's school system, and his call for merit pay of teachers, drew loud applause.

Reagan also called for "restoring parents and local government to their rightful role in the education process."

Earlier in his remarks, Reagan alluded to a more immediate problem for Seton Hall's more than 1,900 graduates: moving from school into the job market during a time of 10 percent unemployment. He told them not to be discouraged.

He was graduated 51 years ago, in the midst of the Depression, Reagan said, and "here we are, a half-century later, and it has been a half-century of ever-increasing opportunity and adventure. Life has been good."