President Reagan harshly criticized Soviet treaty violations today and told Canadian Prime Minister Brian Mulroney that the prospects of a superpower summit are now up to Moscow.

"The ball is in their court," Reagan said, according to a senior U.S. official, who said Reagan was prepared "at this point" for a "substantive" meeting with the new Soviet leader, Mikhail Gorbachev.

Reagan tempered his criticism of Soviet treaty violations by saying he hoped Gorbachev's rise last week "will open up new possibilities" for "more constructive relations."

Officials said Reagan's sharp criticism of Moscow today was intended to represent a "return to realism" toward the Soviets while still holding out the prospects for a summit. He has invited Gorbachev to meet with him, but the Soviet leader apparently has not responded affirmatively yet.

The harmony that Reagan and Mulroney strove to display in their second day of meetings was jarred by a suggestion from Defense Secretary Caspar W. Weinberger that the United States might deploy missile defenses against incoming Soviet cruise missiles on Canadian soil.

The comment, made in an interview with Canadian television, provoked a flurry of attention because of Canada's sensitivity to the placement of U.S. arms here, and White House officials privately said they were irritated at Weinberger for making the remark.

Presidential spokesman Larry Speakes said that "it is a mistake to think the United States has any plans" for such a system and that it would "certainly be a Canadian decision" if it were needed, and Canada's external affairs minister, Joe Clark, said he believed none was necessary.

Meeting on the ice-clogged St. Lawrence River at the historic Citadelle, built to shore up Canadian defenses after the War of 1812, Reagan and Mulroney signed a $1.3 billion agreement to modernize the outmoded "Distant Early Warning" line of radar stations across Canada's north to protect against Soviet bombers.

They also signed a treaty expanding cooperation between law enforcement officials in each country, exchanged protocols of a new Pacific Salmon Treaty and announced that Canada will contribute to U.S. development of a manned space station.

They also issued a statement calling for preparations for another round of international trade talks in 1986 and said they would search for ways to allow freer trade between the two nations.

Reagan received vague support for his Strategic Defense Initiative antimissile system in a joint statement describing it as "prudent" and "in conformity with the ABM Antiballistic Missile Treaty."

The president raised the possibility in his luncheon speech today of "sharing" the SDI with Canada and said, "We very much appreciate Canada's support on SDI research."

But Clark said Canada has not been asked to share in the research and has made no decision on whether to do so. A senior U.S. official said Canada had not been asked to share the costs of the so-called "Star Wars" defense system.

Clark added of the SDI: "Our concerns about the ABM Treaty are profound concerns." Reagan has said research on the antimissile system is permitted by the ABM Treaty.

Reagan has criticized the Soviets for treaty violations often, but in recent months had taken a far more conciliatory approach. Last week, he said he thought the Soviets wanted to try along with the United States to reduce nuclear arsenals.

But today, speaking at a luncheon with Canadian and provincal leaders at the Chateau Frontenac Hotel, Reagan adopted a stern tone.

"As much as we may hope for greater stability through arms control, we must remember that the Soviet record of compliance with past agreements has been poor," he said.

"The Soviet Union signed the Yalta accord pledging free elections, then proceeded to dominate Eastern Europe; they signed the Geneva convention banning use of chemical weapons, SALT II limiting development of new weapons, and the ABM Treaty, but are now violating all three; and they signed the Helsinki accord solemnly pledging respect for human rights but then jailed the individuals trying to monitor it in the U.S.S.R."

Describing his antimissile defense system as a "security shield" that could "someday eliminate the threat of nuclear attack," Reagan said: "It puzzles me to hear the Soviets describe research to protect humanity as a threat to peace. Their protests ring a little hollow. I did some research of my own and found that, in 1967, former Soviet premier Aleksei Kosygin said, 'The antimissile system is not a weapon of aggression, or attack, it is a defensive system.' And the Soviets took his words to heart, and began investing heavily in strategic defense."

A White House official said Reagan's remarks were intended to be "a reminder to everybody" that his critical views of Moscow still hold despite the accession of a new Soviet leader.

A second official, who briefed reporters on the condition that he not be identified by name, said a summit meeting now "would be useful." He added, "In order to meet with the Soviet Union you don't have to share their views" on international issues.

On a day in which Reagan and Mulroney hoped to celebrate their personal camaraderie and U.S.-Canadian relations, Weinberger's remark produced a major distraction for officials from both nations.

The remark came against a backdrop of increasing anxiety in Canada about U.S. plans for deployment of nuclear weapons. The test of an unarmed U.S. cruise missile across Alberta recently sparked protests, and the disclosure of U.S. wartime contingency plans to put nuclear weapons in Canada and other nations intensified that concern.

Reagan and Mulroney vowed in a statement today to "strengthen continental defense." The agreement they signed includes modernization of what will now be called the North Warning System, training of Canadians on AWACS radar planes and possible future forward-basing of U.S. aircraft in Canada.

In an interview on Canadian television, Weinberger said the United States and Canada also are working on improving their defenses against cruise missiles once they are spotted.

Asked if this would involve placing missile launchers in Canada, Weinberger said the United States would try to ". . .locate the best places for defenses. Some might be here, some might be in the United States, some might be at sea. It just depends on where the most effective technical place is for them to be put."

The White House, sensing the political turbulence in Canada that such a statement might create, quickly responded by saying that Canada would have to give its approval for such a system, and none was planned.

National security adviser Robert C. McFarlane said it is "premature" to discuss a strategy of stationing missiles in Canada to intercept cruise missiles. He said missiles are "not the best way" to deal with the Soviet cruise missile, and interceptors are far more likely for now.

The subject of Weinberger's comment came up at a meeting of Reagan, Mulroney and top officials from each nation. Clark told reporters that "Canada retains the sovereign right" to determine whether U.S. weapons would be stationed on its soil.