Sen. Jacob K. Javits (R-N.Y.) lost his bid for the Republican nomination to a fifth term yesterday in the most stunning upset so far in the 1980 congressional primaries.

Javits, age 76 and ailing, ran behind conservative Alfonse D'Amato, a Long Island municipal official, in the first Republican primary challenge of his nearly quarter-century Senate career. With 98 percent of the vote counted from the bitter GOP primary rate, D'Amato led Javits, 56 to 44 percent.

In the state's Democratic senatorial race, Rep. Elizabeth Holtzman (D-N.Y.) won the nomination, defeating former Miss America and consumer advocate Bess Myerson, former New York City mayor John Lindsay and Queens District Attorney John Santucci.

Javits, a leading liberal voice in Congress, has the Liberal Party nomination for the Senate and hence can still run in November on the Liberal ballot line.

Although he had suggested earlier he might not run if he lost the GOP nomination, he said last night he would wage a vigorous campaign this fall and added, "I expect to win reelection in November."

Sharing the Liberal Party endorsement with independent presidential candidate John B. Anderson, Javits may well add to Anderson's vote, which is expected come at the expense of President Carter in New York.

In two races where Abscam defendants were seeking reelection, Rep. Richard Kelly (R-Fla.) was defeated, while Rep. John Murphy (D-N.Y.) was renominated.

In Florida, Sen. Richard Stone (D-Fla.), assaulted from left and right in his bid for renomination, appeared headed for an October runoff primary with State Insurance Commissioner Bill Gunter, whom Stone defeated in a runoff primary in 1974. Rep. Edward J. Stack (D-Fla.) was defeated in his first bid for reelection.

In Connecticut, former New York senator James Buckley won the Republican nomination to run against Democratic Rep. Christopher Dodd in the November race to succeed retiring Democratic Sen. Abraham Ribicoff.

In brief remarks to supporters at a Manhattan hotel, Javits said "recriminations . . . are not in order" and said he interpreted the Republican primary vote "as I have taken every other [experience] in life -- as a means of learning and as an incentive to do better rather than as a repudiation."

Holtzman, flanked by Gloria Steinem, Bella Abzug and others, told an overflow crowd at her campaign headquarters that her come-from-behind victory "shows that someone who, from the beginning, has stood up to the special interests . . . stood up to the political machine . . . can be a winner in New York state."

Holtzman, who campaigned without the support of either Gov. Hugh Carey or New York Mayor Edward Koch, told a cheering crowd at the Halloran House Hotel that her victory also "shows that you don't have to be a man to be a winner in New York state."

"Today is significant because it marks the first time in New York's history that a woman has been nominated for the United States Senate by a major party," she said.

In the only other senatorial upset so far this year, Sen. Mike Gravel (D-Alaska) lost his bid for renomination in that state's Democratic primary last month.

But the defeat of Javits -- a voice of liberalism in a party dominated by conservatives, one of the Senate's keenest intellects and a key link in the bipartisan centrist network that normally runs Congress -- was clearly the biggest upset.

D'Amato, 43, presiding supervisor of the township of Hempstead on Long Island, ran a hard-hitting campaign against Javits, hammering directly at his age and his affliction with motor neuron disease, a progressively crippling nerve disorder that Javits disclosed in announcing for reelection last February.

In New Hampshire, Meldrim Thomson, the conservative former governor who was unseated by the Democrats two years ago, launched a come back drive by winning the Republican gubernatorial primary.

Other primary results: Arizona

Four-term Republican Sen. Barry Goldwater, 71 and suffering from a hip ailment, was unchallenged for renomination in the GOP primary but faces what could be a tough race in November from the victor in yesterday's three-way Democratic primary.

In the Democratic race, businessman Bill Schulz, who put nearly $1 million of his own money into his campaign, was leading former state Senator Jim McNulty, a lawyer who attacked both Goldwater's record and Schulz's spending. Surgeon Frank DePaoli was running a district third. Colorado

Howard (Bo) Callaway was narrowly edged out by Colorado Secretary of State Mary Estill Buchanan in a four-way primary for the Republican nomination, has been tagged by the GOP as a prime target for defeat in November. Callaway is a former Georgia congressman who now operates a ski resort in Colorado.

A recount appeared likely. Connecticut

Former New York senator James Buckley easily defeated state Senate Minority Leader Richard Buzzuto in Connecticut's Republican primary.

Buckley will face Democratic nominee Rep. Christopher Dodd in the November election to fill the seat vacataed by retiring senator Abraham Ribicoff. Buckley represented New York from 1971 to 1977. He was defeated by Daniel Patrick Moynihan.

Buckley, a nationally known conservative, completely ignored Bozzuto throughout the primary and outspent him by 5 to 1.

John Aristotle Phillips, who drew national attention for his undergraduate college thesis on how to design an atomic bomb, won the Democratic nomination for Congress from Fairfield County.

Phillips heads a citizen's lobby promoting solar energy and just turned 25 in August. He will face five-term Rep. Stewart McKinney (R-Conn.) in November. Florida

Rep. Richard Kelly, facing trial next month in the Abscam scandal, was soundly trounced by two Republican challengers in Florida's primary elections. Kelly, who claimed he was conducting his own bribery investigation in the Abscam incidents, trailed Sen. Vince Fechtel, of Leesburg, and attorney Bill McCollum, of Altamonte Springs.

And in one of the toughest Democratic races this year, first term Sen. Richard Stone appeared headed for a runoff against insurance commissioner Bill Gunter. Gunter won the 1974 Democratic primary before lossing a runoff to Stone.

In a Republican field of six, Paula Hawkins, former chairman of the state's public service commission, outdistanced five other candidates. Hawkins held a wide lead over former U.S. representative Lou Frey, but was just short of a majority of the vote.

In other Florida races, freshman Rep. Edward Stack was defeated by 34-year-old attorney Alan Becker. Becker, a state representative, lost a runoff for the Democratic attorney general nomination in 1978. He has representated condominium residents in a district where the condominum population is important.

Oct. 7 is the date for any senate runoff election. Minnesota

Five of seven incumbent House members were unopposed for renomination. In the remaining two contests, Democratic Rep. James Oberstar defeated Duluth City Council member Thomas Dougherty in a much tougher race than had been expected. Rep. Bill Frenzel won handily against a 53-year-old electrician. Nevada

Republican Sen. Paul Laxalt had the good fortune to be opposed in the primary by a candidate who didn't bother to campaign. Laxalt, who is Republican presidential nominee Ronald Reagan's national campaign chairman, will be challenged in November by Former Democratic state senator Mary Gojack, who had no opposition for her party's nomination, and by Libertarian Party candidate Allen Hacker. New Hampshire

Meldrin Thomson, the controversial former governor who once advocated arming the state's National Guard with nuclear weapons, coasted to an easy victory in the Republican primary for governor.

The conservative Thomson, seeking an unprecendented fourth term in the New Hampshire statehouse, was defeated in 1978 by Democrat Hugh J. Gallen, who now is seeking reelection.

In the 11-person GOP Senate contest, former state attorney general Warren Rudman took a big early lead in the race to challenge Democratic Sen. John Durkin.

In the 1st Congressional District, former Manchester mayor John Mongan lead in the race to run against incumbent Democrat Norman D'Amours, who had no primary challenge.

In the 2nd Congressional District Republican contest -- where incumbent Republican James Cleveland is retiring -- Executive Councilor Judd Gregg was running well ahead of state Sen. Susan McLane. Rhode Island

Democratic Gov. J. Joseph Garrahy and his Republican challenger, Vincent A. Cianci Jr., had no primary opposition. Two Rhode Island congressmen, Rep. Edward Beard and Rep. Fernand St. German, comfortably beat back challenges to their renominations. Utah

Republican Sen. Jake Garn was unopposed for renomination and is heavily favored for reelection, but two Democrats -- A. Stephen Dirks, mayor of Ogden, and Dan Berman, a Salt Lake City attorney -- were in a close race for the nomination to oppose him in November. Vermont

Stewart Ledbetter, who walked the length of Vermont to demonstrate frugality in contrast to his heavy spending opposition, won a six way primary for the Republican nomination for the Senate yesterday. Ledbetter will face incumbent Patrick Leahy in November. Leahy is the only Democrat Vermont has ever sent to the Senate, and he is considered a prime GOP target in the coming election.

Ledbetter resigned as state banking and insurance commissioner earlier this year. His principal opponent, James Mullin, alone spent $700,000. Mullin, a self-proclaimed "clone of Ronald Reagan," waged the most expensive campaign in the state's history.

In the Democratic gubernatorial primary, House Speaker Timothy O'Connor and Attorney General M. Jerome Diamond were neck and neck in their race to run against Republican Gov. Richard Snelling, who is seeking reelection. Wisconsin

Former representative Robert W. Kasten Jr., won a four-way Republican primary yesterday and the opportunity to run against Democratic Sen. Gaylord Nelson in November. Nelson faced no primary opposition.

Democratic Reps. Clement Zablocki and Henry Reuss, both from Milwaukee, won easily over token opposition. Wyoming

Republican Rep. Richard Cheney, White House chief of staff in the Ford administration, was unopposed for renomination. Jim Rogers won the Democratic primary for the state's only congressional seat, outdistancing three other candidates, none of them well-known.