Australian voters have thrown out the conservative government of Malcolm Fraser and elected the mildly socialist Labor Party led by Robert Hawke to lead the country for the next three years.

Hawke was swept to power yesterday in a landslide that saw Fraser's 21-seat majority in the 125-member national House of Representatives swing to a Labor Party majority expected to be between 17 and 27 seats. Five races remained undecided.

Close to tears, Fraser immediately quit as leader of the defeated Liberal Party and said he would not contest the leadership again.

The reins of conservative leadership in Australia are now likely to go to Andrew Peacock, who was foreign minister from 1975 to 1980 and who has long been a rival of Fraser.

Like Hawke, Peacock is a charismatic politician given more to the style of American campaigning than to the dour arguing of economic issues that marked Fraser's leadership.

The win for Hawke capped the most spectacular rise to political power since Australia became an independent federation in 1901.

He rose to fame and achieved great popularity as the president of the Australian Council of Trade Unions, the equivalent of the AFL-CIO, during most of the 1970s. A brilliant, irascible performer on television, famous for his vigorous attacks on interviewers, Hawke was also the most persuasive orator in the country.

He did not enter politics until 1980, when he won a seat representing a "safe" Labor district in the federal Parliament. He had greater official political standing there than any other freshman legislator.

The Labor Party leader then was a former policeman from Queensland, Bill Hayden, who ran a close second to Fraser in the 1980 election. Hawke supporters in the Labor Party pushed for their man to become leader, although it was against the strongest tradition of the party, the oldest in Australia, to replace a leader until he volunteered to resign.

Hawke challenged Hayden for the leadership last July, but lost in a narrow vote of party members.

On Feb. 3, the day Fraser called an election nearly a year earlier than he needed to, Hayden resigned as leader in Hawke's favor. Hayden made the move reluctantly and under intense pressure from Labor power brokers who saw Hawke as the man who would lead them into power.

Hawke campaigned brilliantly. His rallies were huge and his oratory more of the style of an Edward M. Kennedy than any other Australian politician in recent years. He presented a policy more socialistic than any offered to the Australian public in modern times.

Its centerpiece is a prices and incomes policy that includes an agreement with the trade unions that they will not press for big wage increases in return for Hawke implementing many programs cherished by the unions. They include a ban on foreign banks setting up in Australia, stronger controls on multinational corporations, price controls, restrictions on the profit levels of corporations and the incomes of non-wage earners such as professional people and small businessmen.

In the campaign, Fraser lashed out at Hawke for trying to impose socialism on Australians and for offering to give too much power to the trade unions. He declared that the people's savings would be safer under the bed than in banks with a Hawke government because Labor would impose more controls on financial institutions.

The comment backfired badly. A major theme of Hawke's campaign was that the public knew him well and that he could be trusted to bring Australians together to reach a consensus to lift the nation economically and make it more just.

The public did not believe Fraser's comment on their savings, there was no run on the banks and Fraser was seen as a man who had panicked in the face of defeat, thus losing his greatest political strength--the image of steadiness and reason that had kept him in power for more than seven years.

Hawke is likely to modify some of the more radical plans in the Labor platform. He has many friends in the business world and says he is a strong believer in private enterprise.

In foreign policy, he is a strong backer of Australia's alliance with the United States and has long been an emotional supporter of Israel.

Reuter news service quoted White House deputy press secretary Larry Speakes as saying President Reagan had sent Hawke a congratulatory telegram and pledged "to continue a close and effective relationship with the new government."

But foreign policy is unlikely to be a major part of the new government's agenda for the next several months. Hawke's first plan is to call a national economic summit of government, union and business leaders to achieve a consensus on how to solve Australia's problems of high unemployment, high inflation and no growth.