The Senate Judiciary Committee, by a vote of 9 to 5, recommended yesterday that the Senate reject the nomination of Judge Robert H. Bork to the Supreme Court, increasing pressure on President Reagan to concede defeat and withdraw his name.

Sen. Howell Heflin (D-Ala.), who had been uncommitted, joined the committee's seven other Democrats and Sen. Arlen Specter (R-Pa.) in voting to send the nomination to the Senate floor with an unfavorable recommendation. White House officials had seen Heflin as their last hope of persuading other uncommitted southern Democrats to buck the strong tide against Bork and support confirmation.

Senate Republicans and administration lobbyists, however, vowed to continue the fight and force a vote by the full Senate even though Bork's cause now appears hopeless. Assistant Attorney General John R. Bolton said that he spoke with Bork Monday and that "Judge Bork wants to go to the floor."

Administration officials also indicated that by forcing a vote in the full Senate, they hope to inflict political damage on conservative, southern Democrats who oppose Bork. {Details, Page A14.}

Sen. Gordon J. Humphrey (R-N.H.) urged Reagan "to continue to stand firm because we shall win one way or another -- either by confirming Robert Bork or by making this process a political issue in 1988 and beyond."

Senate Democrats dismissed the possibility that they will suffer politically from the bruising confirmation fight.

"I think it is a losing issue for them {Republicans}," Senate Majority Whip Alan Cranston (D-Calif.) said. "A majority of the people do not want Bork."

Judiciary Committee Chairman Joseph R. Biden Jr. (D-Del.) added, "I said from the beginning the White House has so misunderstood where the American people are on these fundamental issues, and they seem not to have learned the lesson."

Senate leaders agreed yesterday that the nomination will be ready for floor debate no later than Oct. 16, but it could begin early next week. Senate Majority Leader Robert C. Byrd (D-W.Va.) has said that if Reagan does not withdraw the nomination he will force a vote on the issue as soon as possible.

Yesterday's Judiciary Committee vote matched the highest expectations of Bork's opponents. All four of the committee members who began the confirmation process undecided -- Byrd, Heflin, Specter and Sen. Dennis DeConcini (D-Ariz.) -- voted against confirmation.

Heflin, the last to make up his mind, waited until the final sentence in a 10-minute statement to declare himself. He said a lifetime appointment to the Supreme Court "is too important to risk to a person who has continued to exhibit, and may still possess, a proclivity for extremism in spite of confirmation protestations."

Heflin hedged his statement by saying it applied "at this time and at this posture of the confirmation process." After the committee meeting, he said he could still change his mind. Biden, suggesting that Heflin's vote is no longer necessary to defeat Bork, said he was not concerned about a possible reversal.

Earlier yesterday, a fifth Republican, Sen. Robert T. Stafford (Vt.), announced that he will vote against confirmation. Cranston said his latest vote count showed 54 votes against confirmation, 42 for and four senators undecided.

The ornate Senate Caucus Room was jammed with reporters, lobbyists and spectators as the Judiciary Committee began yesterday's meeting with 90 minutes of prepared statements by the panel's eight Democrats and six Republicans. Appearing dispirited by the certainty of defeat, committee Republicans attacked the intense public campaign waged against Bork by a broad coalition of civil rights, women's and other liberal groups.

Sen. Charles E. Grassley (R-Iowa) said he was ashamed of the "distortions" and "mob justice" to which Bork had been subjected during the 12 days of committee hearings. Bork's opponents, he said, "had transformed high-minded constitutional debate into the worst pressure-group politics I've ever seen."

Senate Minority Whip Alan K. Simpson (R-Wyo.) called the confirmation process "very demeaning" and, quoting 18th-century British statesman Edmund Burke, said Bork had been rejected by "the presumptuous judgment of the ignorant."

Biden replied to these charges by saying that to credit the lobbying campaign for Bork's troubles "in a sense undersells and undercuts the wisdom of the American people."

Calling Bork "a man of honor, integrity and intellect," Biden said the debate was "not about Judge Bork; it's about the Constitution."

"Do I have certain inalienable rights because I exist, or do I have rights because my government confers them on me?" he said. "Judge Bork's view is the latter; mine is the former. I never had any doubt where the public would come down."

Simpson said he talked to Bork last Thursday and quoted the 60-year-old jurist as saying: "Did you call to cheer me up? Don't feel any anguish or despair for me. I'm a big boy and I know full well what's happening. I have a fine job, and I'll be doing it the rest of my life if I don't go to the Supreme Court."

Bork is now running against not only a strong political tide in the Senate, but historic precedent. The Judiciary Committee has rejected three other Supreme Court nominations and sent one nomination to the floor with no recommendation. All four nominations were defeated by the full Senate.

Reagan nominated Bork to succeed retired justice Lewis F. Powell Jr. on July 1. After a 25-year career of prolific writing and speaking, Bork was well known to political activists of both the left and the right and his nomination set off a frenzied effort by liberal opponents to block confirmation.

The campaign against Bork made use of many of the techniques of modern politics, including public opinion polling, radio and television commercials and mass mailings. But the key event during the fight was probably Bork's five days of televised testimony before the Judiciary Committee last month in which he clearly failed to quell fears that his earlier, harsh criticism of several landmark Supreme Court decisions reflected the attitude he would bring to the court.