In a stunning upset, California Sen. Alan Cranston defeated Democratic presidential front-runner Walter F. Mondale in a straw poll of delegates to the Wisconsin Democratic Convention today, significantly boosting Cranston's long-shot candidacy.

Cranston won on the nuclear-freeze issue that has become his campaign's centerpiece. He finished with 38.8 percent of 2,035 votes cast, compared to Mondale's 35.7 percent and Colorado Sen. Gary Hart's 21.8 percent.

The defeat startled Mondale strategists and could cause significant damage to Mondale's candidacy.

The former vice president's advisers had confidently predicted victory in this straw poll that has no official bearing on the delegate selection process for 1984. Gov. Anthony S. Earl (D), a Mondale supporter, had predicted that Mondale would win 50 percent of the vote.

Mondale, from neighboring Minnesota, had the support of most of Wisconsin's prominent Democratic officials. However, Cranston had set his sights on Wisconsin early, addressing a nuclear-freeze rally in September, campaigning in the state frequently and attracting endorsements from several of the most liberal activists in a state with a long progressive and liberal tradition.

Mondale's defeat was good news for Ohio Sen. John Glenn, a middle-of-the-road Democrat who has been gaining strongly on Mondale in national opinion polls and decided not to appear at the convention rather than risk an embarrassing defeat.

Final poll totals, according to state party officials, were: Cranston 789 votes, Mondale 727, Hart 443, Glenn 39, former Florida governor Reubin Askew 14, South Carolina Sen. Ernest F. Hollings 1 and undecided 22.

Hart's forces, who had been locked in a battle with Cranston to be a liberal alternative to Mondale, felt they had staved off what would have been a devastating defeat with a strong push in the last week that added to their candidate's vote totals.

Cranston jubilantly proclaimed that his victory "shows that I am one of the top three contenders. I think there are only three of us in the new tier . . . Mondale, Glenn, Cranston."

Cranston said that he had not expected to win the straw poll but thought he had an outside chance.

Mondale appeared stunned and was at times testy as he spoke to reporters after his defeat. "As you know I've had many many good days in this campaign for president," Mondale said. "This was not one of them." He said that he plans to make no changes in his campaign or his message. "What happened here, I'm not sure," he said.

Hart said that he figures that the race is now four-way, with him very much in the first tier of candidates. He said that he does not expect any major breakthroughs in his candidacy until 1984. The straw poll, he said, shows what he has been saying all along, that "the race is wide open. Front-runnership in the spring of 1983 means nothing."

Meanwhile, out in the corridor, Cranston was happily bubbling, "I disclaim all assertions that I'm the front-runner."

Cranston and Hart had lifted their very personal battle out of the political trenches and into klieg lights here.

Cranston told a news conference Friday that Hart approached him recently on the Senate floor to protest angrily what he felt were rumors being spread by the Cranston camp about Hart's candidacy being in deep distress. According to Cranston, Hart said that he wanted it stopped. Cranston said he then told his aides to stop talking about Hart, if indeed they had been.

Today, Hart and Cranston used their convention speeches to discuss each other, carefully naming no names but certainly leaving no doubt. Each aimed a not-too-subtle salvo at the heart of the other's appeal.

Hart, who spoke first, sought to crack Cranston's staunch bloc of nuclear-freeze supporters, the core of Cranston's candidacy, by reminding them that Cranston supported the B1 bomber. Hart said, "You and I . . . know that, as much as we need the freeze, we also need to stop production of the MX missile and the B1 bomber--now!"

Hart also jabbed Cranston's continued acceptance of contributions from special-interest political action committees, saying: "I've heard the voices of Wisconsin. . . raised against the money lenders in the temple of Democratic politics. . . . But you and I also know that the only way to break the stranglehold is for no Democratic candidate to take special-interest money."

Later, Cranston got in his digs. He jabbed at Hart, who has wrapped his campaign around the notion that, as a leader of a new generation of Democrats known as neo-liberals, he will provide new ideas to solve the problems of a technological age.

"High-tech is no magical solution," Cranston said. "I'm no Atari Democrat. We do need to use new technologies to expand work opportunities. But I don't believe we can abandon our basic industries, steel, rubber, machine tools and autos."

Mondale, meanwhile, steered a high road across the convention battlefield, attacking President Reagan liberally and pausing just long enough to touch a few local monuments.

"I'm a preacher's kid who grew up in our rich midwestern soil," he said. "I was raised with the name of Bob La Follette in my ear and the hand of Hubert Humphrey on my shoulder."

He said that Reagan campaigned in 1980 on five themes--family, work, neighborhood, peace and freedom--and added: "The trouble is that once he won, Reagan ran every one of them into the ground."

Glenn's surrogate speaker, his wife, Annie, spoke with grace and polish, impressive in light of the fact that until a few years ago she was afflicted with a severe stutter.

"John Glenn asked me to tell you he will campaign hard in Wisconsin," she said. She added that she hopes that Democrats campaigning for the presidency will concentrate on attacking Reagan and not "criticize each other . . . emphasizing our differences."

Cranston became the subject of recent press criticism that shook some of his supporters after it was reported that he had spent $4,000 to pay for 100 hotel rooms for some of his delegates who could not afford accommodations.

Mondale and Hart delegates showed up wearing badges that said, "I paid my own way."