Robert E. Short, a Minneapolis businessman who challenged the traditional liberal doctrines of his party and financed his own $800,000 campaign, won an upset primary victory over Rep. Donald M. Fraser (D-Minn.) yesterday for the Democratic nomination for the Senate seat once held by the late Hubert H. Humphrey.

Short, a trucking and hotel magnate who owned the Washington Senators baseball team before moving to Texas, then selling it, held a 4,000-vote lead, with more than half-a-million votes counted in a primary that seriously divided the Minnesota Democrat-Farmer-Labor Party (DLF) and left its prospects for the November election uncertain.

Fraser, who had gone to bed Tuesday night thinking himself the victor, declined to concede defeat or endorse Short, pending an official canvass of the vote, which is to begin tomorrow.

The 16-year House veteran, endorsed by his party's convention, by Vice President Mondale and by Humphrey's widow, interim Sen. Muriel Humphrey (D-Minn.), had piled up a 30,000-vote lead early yesterday morning on returns from the Minneapolis-St. Paul metropolitan area.

But Short reversed the trend in conservative and Catholic rural areas, where his own antiabortion stand was far more popular than Fraser's support of abortion rights.

And Short, a longtime Humphrey financial backer, buried Fraser in the northern Minnesota Iron Range, where Fraser's efforts to ban motorboating and snowmobiling in the Boundary Waters Canoe Area had drawn strong criticism from resort owners, sportsmen and others.

There was evidence that Short's open appeal to Republicans to cross over and vote against the liberal Fraser had had some effect. About 72 percent of the votes - an unusually high percentage - were cast in the DFL primary. And a telephone poll taken for The Minneapolis Star indicated that almost one-fifth of the voters in the DFL primary considered themselves Republicans.

Minnesota is one of several states that allows Democrats to vote in Republican primaries, and vice-versa.

In a news conference yesterday, Short said he would continue to press the same antispending, anti-tax issues he had used against Fraser in the contest with GOP nominee David Durenberger, a moderate Republican who was an easy winner in a five-man GOP primary.

Saying that voters were tired of the welfare-state liberalism long associated with minnesota politicians like Fraser, Short called for a $100 billion cutback in federal spending and a $50 billion tax cut as the best way to combat inflation.

The Minneapolis Star poll found those taxing spending issues ranked as most important by DFL primary voters, but also noted that 40 percent of those who voted for Short said his antiabortion position was one of the most influential reasons for their choice. Newspaper ads and flyers hit the abortion issue hard on the final weekend of the campaign.

Durenberger, a Minneapolis lawyer making his first bid for major office, had positioned himself to run against Fraser. in a preprimary interview, he noted the similarities between his own and Short's views on abortion, the Boundary Waters recreation area, national defense and federal spending, and predicted, "I will have a good crack at most of the Short voters."

Yesterday, Durenberger said Short's nomination might mean "less of an opportunity to debate specific issues," but said he was heartned by the number of Fraser supporters offering to help him, because, "Even if they don't agree with me on all the issues, they know I understand the role of government and will offer a more positive approach than Bob Short did."

Privately, many DFL leaders said they expected Fraser's supporters - many of them longtime party activists - to sit on their hands in the general election.

"You can't beat the endorsed candidate in the DFL and win the election," said one DFL leader. He noted that in 1966, when Short was the lieutenant governor candidate on a ticket that upset the endorsed DFL team, Republicans took over the state government.

But Short's campaign manager, Fred Gates, said that Short would win in November by appealing to "independent voters," as he had in the primary, and predicted that organized labor, which backed Fraser as the endorsed candidate, would lend its support to Short.

The upset of Fraser left Sen. Wendell R. Anderson (D-Minn.) an isolated defender of the traditional Humphrey-style liberalism of the Minnesota Democrats. Anderson, who resigned as governor last year and had himself appointed to Mondale's Senate seat, breezed past his primary foe, John S. Connelly. He now faces plywood industry millionaire Rudy Boschwitz, winner of the GOP nomination for the second Senate seat.

The gubernatorial battle pits incumbent Gov. Rudy Perpich (D) against Rep. Albert H. Quie (R-Minn.), who has been in the House for 20 years.

Preprimary polls showed the Perpich-Quie race a tossup and Anderson trailing Boschwitz, largely because of the continuing controversy over his "self-appointment" to the Senate.

Unofficial returns from all but 10 of the 4,019 precincts gave Short 257,377 votes and Fraser 253,419.