The Senate Judiciary Committee refused to approve one of President Reagan's judicial nominees for the first time yesterday, then agreed in a second vote to send the nomination of Daniel A. Manion to the floor without a recommendation.

After hearing impassioned arguments that the Indiana lawyer is too inexperienced and his views too extreme, the panel rejected his nomination to a federal appeals court on a 9-to-9 tie.

Key negative votes were cast by Republicans Arlen Specter (Pa.) and Charles McC. Mathias Jr. (Md.) and Democrat Dennis DeConcini (Ariz.). A moment later, however, Specter and DeConcini switched their votes on a motion to have the full Senate consider the nomination without committee approval.

Democrats said the battle over Manion is a test of the Senate's will to block Reagan judicial nominees who Democrats contend are marginally qualified. "This is one we'll go to the wall on," ranking committee Democrat Joseph R. Biden Jr. (Del.) said.

Manion, 44, a former state senator, is the son of the late Clarence Manion, a John Birch Society leader and former dean of Notre Dame Law School. The nominee frequently appeared with his father on a syndicated broadcast called "The Manion Forum" in the 1970s, making controversial statements that his critics have cited at length.

Democrats said they fear that Manion would follow his conservative ideology, not court precedents, if confirmed for the 7th Circuit Court of Appeals in Chicago.

They took pains yesterday to emphasize what they view as his meager record, including a small law practice in South Bend that handles mainly personal and commercial claims in state court.

"Not once has he had the prime responsibility in a federal trial," Sen. Edward M. Kennedy (D-Mass.) said. "He never handled a case in which a federal constitutional issue was involved, never once, and never argued a case in the 7th Circuit."

Committee Chairman Strom Thurmond (R-S.C.) dismissed such arguments, saying: "It is obvious most of the criticism leveled against Mr. Manion is strictly ideological in nature. He has assured the committee he will follow the Constitution and the decisions of the Supreme Court. I am confident he will make a fine circuit court judge."

Biden retorted that he had voted to confirm such conservative appeals court judges as Antonin Scalia, Robert H. Bork and Richard A. Posner.

"These and other acknowledged conservatives had . . . the earmarks of excellence," he said. "Quite frankly, Mr. Manion is not in their class. His academic distinction and scholarship are similar to my own, and I wouldn't put me on the bench."

In a long, emotional speech, Biden also said he was skeptical of "recantations" from nominees about their past records.

"We have been too accepting of the rights of nominees to explain over and over again what they meant or did not mean when they made certain statements," he said. " . . . To suggest that my opposition and that of others is based on ideology is in fact a smokescreen, because it masks the nominee's patent lack of qualifications and uncertainty about whether he will uphold the Constitution."

Kennedy read at length from transcripts of broadcasts in which Manion said the Supreme Court had gone too far in rulings on school prayer and pornography and should be stripped of some of its jurisdiction. He also quoted Manion's criticism of the doctrine that courts have used for 60 years to apply key Bill of Rights protections to the states.

Kennedy also accused Manion of having "a selective memory" about his past statements, such as a 1979 letter praising John Birch Society members for being "on the front line of the fight for constitutional freedom."

Kennedy and Biden charged Manion with violating his oath of office in the Indiana Senate by cosponsoring a 1981 bill to allow posting the Ten Commandments in public schools. Manion acknowledged that he knew the Supreme Court had ruled the practice unconstitutional two months earlier.

Reagan, who has named more than one-third of the nation's federal judges, has attempted to recast the judiciary in a conservative mold. Democrats have criticized a few, such as Jefferson B. Sessions III, nominee to a district court judgeship in Alabama, who is alleged to have made racially insensitive remarks.

The quality of Reagan's nominees has become a growing political issue in recent months, and the Manion debate is the first to focus on whether a nominee can be rejected for being too far out of the political mainstream.

Sen. Orrin G. Hatch (R-Utah) said that he voted for hundreds of judges in the Carter administration whose philosophies he found "repugnant" and that Manion should not be rejected for statements he made as "a young man."

Liberal groups ranging from the Leadership Conference on Civil Rights to the National Organization for Women waged an all-out campaign against the nomination.

"If the Senate approves Manion, there is no one they will not hold their nose and vote for," said Nancy Broff of the Alliance for Justice.

DeConcini said he opposed Manion because of his lack of experience, while Specter cited the nominee's views on the Constitution. Sen. Howell Heflin (Ala.) was the only Democrat voting for confirmation.