Secretary of State Cyrus Vance wound up two days of discussion with Canadian leaders today, saying he had made substantial progress in resolving key irritants in U.S.-Canadian relations.

In talks with Prime Minister Pierre Trudeau and Foreign Minister Don Jamieson. Vance reached agreement on setting a Dec. 31 deadline for the resolution of the U.S. Canadian maritime boundaries and fisheries issue. Should the negotiators fail to reach an accord, the matter may be turned over to third party arbitration.

Vance said the Canadian leaders have expressed "strong support" for moving ahead with a $10-billion dollar pipeline to carry natural gas from Alaska and Canada's western provinces to the United States.

Vance and Barbara Blum, deputy administrator of the U.S. Environmental Protection Agency, also signed a Great Lakes water quality agreement that Blum called "an effort to reverse a century of abuse and neglect of the largest body of fresh water in the world."

In a speech Tuesday night, Vance made a strong appeal for Canadian unity, in apparent reference to the current Canadian national crisis caused by the threat of the separatist government of Quebec to seek sovereignty for the French-speaking province.

"All of us hope that this great rich country will remain united," Vance said to an audience of top government and business figures. "The decision is one for which Canadians must find their own solution. All Americans want to continue working and living together with a united Canada as our best and our closest friend."

Vance's strong endorsement of Trudeau's policies on national unity was matched by the exceptionally warm welcome the secretary of state received here. In contrast to public rhetoric, however, American and Canadian negotiators were reportedly involved in hard private bargaining on bilateral matters, especially those dealing with energy.

The Americans have been annoyed by Canada's handling of the Alaska pipeline project, especially by what appeared to be a tendency to favor Canadian firms with major contracts. One of Vance's aims was to correct this trend in favor of more a equitable share of contracts for American firms.

Moreover, Vance underscored U.S. desire to begin early construction of the southern portion of the pipeline that would lead from the oil and natural gas rich province of Alberta into the western United States and northern tier states.

The Americans emphasized that tha pipeline construction and the availability of huge natural gas reserves in Alberta would enable project sponsors to raise capital for the construction of what would be an enormous private business venture.

The Canadians, however, fear that the construction of the southern leg and the availability of Alberta gas may diminish the investor's determination to continue the construction of the nearly 4,000-mile portion of the pipeline across the frozen Arctic part of Canada and into Alaska.

The proposal to turn the boundaries dispute over to arbitration reflects an impasse reached by U.S. and Canadian negotiators in the dispute, which originated in the extension of offshore economic zones of the two countries to 200 miles.

Both sides have had to contend with public pressures involving powerful fishing interests and with the prospect of the discovery of oil and gas deposits in the offshore areas.

The Great Lakes water control agreement is in fact an updated version of the 1972 accord. But it includes new provisions dealing with pollution caused by forestry, animal husbandry and farming, and it sets absolute limits on the amounts of radio-activity in waters near nuclear generating plants.

A joint statement on maritime boundaries and fisheries issued at the end of Vance's visit here said Vance and Jamieson had approved a plan to work out a formula for the resolution of the boundary dispute in the Gulf of Maine. It also said that the countries would create cooperative management of their Atlantic fisheries within the 200-mile zone.