TORONTO, OCT. 1 -- Ontario's first socialist provincial government was installed here today, raising hopes among New Democratic Party leaders that they can expand the party's growing popularity over the next three years and overturn Prime Minister Brian Mulroney's Conservative national government.

While socialists have ruled at the provincial level in Canada's western and Prairie regions, the New Democrats ascent to power in Ontario -- the country's industrial and financial heartland and its most populous province by far -- was hailed by party officials as a "new era" in Canadian politics.

However, the New Democrats, who have projected the cost of their planned social programs over the next two years at $3.7 billion, are inheriting a $610 million deficit from the outgoing Liberal Party government, a deficit that officials project will reach at least $1.4 billion and possibly $2.6 billion by next year. Ontario's annual budget is $44 billion.

The new provincial premier, Bob Rae, a boyish-looking 42-year-old Rhodes scholar, seemed undaunted by the fiscal projections, declaring after he was sworn in, "If I believed in forecasting, I wouldn't be here."

Indeed, Rae's upset victory in the Sept. 6 provincial elections came as a surprise even to most New Democrats, who, while long aspiring to power, had expected to remain in distant opposition. In a repudiation of the Liberal government led by David Peterson, 37 percent of the provincial electorate voted for the New Democrats, enough to give them 74 of the legislature's 130 seats and to become the first Ontario government elected with less than 40 percent of the vote. The Liberals won 32 percent of the vote, and the Progressive Conservatives 23 percent.

Peterson called for the election two years before his five-year electoral mandate expired -- a decision political analysts ascribed to Liberals' hopes of renewing their hold on power in advance of an expected economic downturn and before a prominent party fund-raiser could be tried on charges of diverting charitable funds to political uses.

The Liberals' defeat has been widely interpreted by political analysts as not so much a turn toward socialism by Ontario voters as a backlash against what many viewed as cynical and unresponsive government.

In the context of Canada's deeply entrenched social-welfare system, the New Democrats' brand of socialism is relatively benign, and Rae has gone out of his way to reassure the business community that he plans no radical reforms that would adversely affect their interests.

It was popular disaffection with the old back-room style of politics that Rae appeared to address on his first day in office, naming to his 26-member cabinet 11 women and nine persons with no legislative or government experience. Rae also added a populist flavor to the inauguration by staging it at the University of Toronto and opening it to the public, rather than following precedent with a traditional, closed ceremony at the Queen's Park legislature.

In his inaugural speech, Rae appeared determined to walk a tightrope between nervous Ontario businessmen and his labor union backers. "We are committed to provide economic leadership that is responsible, that draws on the strengths and abilities of all communities," Rae said. "To labor and business, I say we must plan together and work together to build recovery. . . . No government can spend its way out of a recession, but that is no excuse for inactivity."

But he made clear at the same time that his party does not mean to abandon its principles of social democracy. On the issue of government priorities, Rae declared: "We are going to have to wrestle with this question, bearing in mind that it is the needs of the people that come first and foremost as we build this partnership."

Among programs promised by Rae are higher minimum wages and tax breaks for low-income families, a minimum corporate tax, inheritance taxes on large estates, a doubling of the budget for home-care nursing, creation of a government-run automobile insurance plan, and a freeze on development of nuclear power plants.

Rae, a longtime opponent of the U.S.-Canada free-trade agreement, also has promised new legislation to assist workers adversely affected by the pact. Such a measure is likely to draw little criticism, since a Gallup poll released today shows that only 5 percent of Canadians believe that Canada has gained more from the agreement than the United States, while 71 percent believe the United States has gained more.

The installation of the New Democrats in Ontario comes at a time when the party's popularity is on the upswing nationally, according to opinion polls. A survey conducted in mid-September for the Southam newspapers indicated that the party is favored by 33 percent of decided voters, while support for the Liberals dropped from 50 percent to 39. Only 18 percent said they would vote for the Progressive Conservatives.

Political analysts have suggested that if Prime Minister Mulroney's ruling Conservatives fail to recover their popularity by the 1993 national election, the New Democrats could win enough votes to join a minority coalition government with the Liberals.