The Senate Judiciary Committee rejected one of President Reagan's judicial nominees for the first time yesterday, voting down Jefferson Beauregard Sessions III amid accusations that he has demonstrated an insensitivity to civil rights.

The 10-to-8 vote by the committee, which has approved 269 Reagan nominees to the federal bench, appeared to signal a new determination by Democrats to block potential judges they regard as too extreme or unqualified. Ranking Democrat Joseph R. Biden Jr. (Del.) said the quality of judges was much higher in Reagan's first term and has become "deplorable" since Attorney General Edwin Meese III took office last year.

But Meese called the vote "an appalling surrender to the politics of ideology." He said Sessions was "an excellent nominee" to the federal district court in Alabama, and blamed his defeat on "vicious and highly personal attacks" by liberal groups "with a big ideological ax to grind."

Despite an emotional plea by Sen. Jeremiah Denton (R-Ala.) on Sessions' behalf, his defeat was assured when his other home-state senator, Howell Heflin (D-Ala.), announced that "my conscience is not clear and I must vote no."

The panel first voted 10 to 8 to reject the nomination, then failed on a 9-to-9 tie to send his name to the Senate floor without a recommendation. Committee Republicans used the no-recommendation strategy last month to salvage an equally controversial appeals court nominee, Daniel A. Manion of Indiana, after failing to muster enough votes to approve him.

Testimony that Sessions, the U.S. attorney in Mobile, had described the NAACP and American Civil Liberties Union as "un-American" helped persuade two Republicans, Charles McC. Mathias Jr. (Md.) and Arlen Specter (Pa.), to join the panel's eight Democrats in opposition. This was the same bipartisan coalition that rejected William Bradford Reynolds' nomination as associate attorney general last year.

Specter switched on the second vote, as he had with Manion, and agreed to forward Sessions' name without a recommendation, leaving Heflin to cast the deciding vote.

Sen. Edward M. Kennedy (D-Mass.) said the vote sends "a clear signal to the Reagan administration that their judicial nominees must meet at least a minimum standard of sensitivity" on civil rights.

The White House was subdued about the defeat, with spokesman Larry Speakes saying that "the Senate has worked its will, and we will certainly have to accept its will."

But Meese issued a combative statement that blamed the defeat on "several liberal organizations" that oppose the Reagan administration. Such groups as the Leadership Conference on Civil Rights, People for the American Way, Alliance for Justice and NAACP Legal Defense Fund led a concerted campaign against Sessions.

Meese called Sessions "the unfortunate victim of people whose tactics are reprehensible and who appear willing to smear anyone in order to advance their agenda, and whose claims of principled opposition are belied by ad hominem attacks that any reasonable person would find repulsive."

Biden responded in an interview that Meese's "vehement reaction, as compared to the White House, is evidence of the fact that he is the ideologue in this administration. He is the one driving this selection process along ideological lines. He and not the Senate is guilty of demanding ideological purity."

Biden said Sessions was rejected solely because of his insensitivity to race issues. He said the vote would aid opponents in the upcoming floor fight over Manion because Republicans have been asked too many times "to walk the plank for people who are unqualified."

Sessions, 38, was nominated last fall after his unsuccessful prosecution of Albert Turner, a former aide to the Rev. Martin Luther King Jr., on voting fraud charges.

This was the first of a string of Justice Department ballot-tampering cases against black civil rights activists in rural Alabama, and it sparked charges, denied by Sessions, that he had selectively prosecuted black leaders.

Sessions' prospects dimmed considerably when his confirmation hearings opened in March and the Democrats, citing affidavits from Justice Department officials, accused him of making a series of racially insensitive remarks.

Sessions acknowledged many of the remarks, but said they were distorted or taken out of context. He maintained, for example, that he had said that some people regard the NAACP and ACLU as "un-American" when they stray into foreign policy and other issues.

Sessions denied testimony that he had called a black assistant "boy." But he acknowledged that he had agreed with another person's statement that a white civil rights attorney was "a disgrace to his race."

Sen. Howard M. Metzenbaum (D-Ohio) said that Sessions "changed his statements" at another hearing last month because "he fully understood that these statements might cost him his judgeship."

On the "disgrace to his race" remark, Metzenbaum said: "Once he said he said it, once he said he didn't recall saying it, and once he said he didn't say it . . . . This is a man who is hostile to civil rights organizations and their causes."

But Denton, who recommended Sessions for the judgeship, charged that Metzenbaum had "recited a litany of misleading statements which do not represent the truth about what Mr. Sessions said or what Mr. Sessions has done." Denton said the media had focused on "inflammatory" allegations and paid little attention to "the refutations, recantations and denials."

Denton, who is running for reelection, said there was "a political conspiracy" in Alabama to discredit Sessions.

The packed hearing room grew quiet when Heflin, a former federal judge, said softly that "this is not an easy vote for me." He said he had "read and reread the transcript" and was troubled by "the alleged inconsistencies" in Sessions' testimony.

"There are admissions, explanations, partial admissions, statements about jokes," Heflin said. He said he had concluded that "a person should not be confirmed for a lifetime appointment as a district judge if there are reasonable doubts about his ability to be fair and impartial."

Yesterday's vote was significant because Reagan is on his way toward naming more than half the nation's federal judges by 1988, and has made no secret of his attempt to recast the judiciary in a conservative mold. Debate may begin next week on Manion, the son of the late John Birch Society leader Clarence Manion, whom critics have accused of being too inexperienced and too far outside the political mainstream.