Canadian air traffic controllers, in unexpectedly abandoning their boycott of U.S.-bound flights early today, appeared to have been overwhelmed by the complaints of the public and a government determined to make a stand against Canadian public-sector strikes.

Among their concerns was believed to be the the possibility that they themselves will strike next month.

By the time the agreement to end the boycott was initialed, Canadian sympathy for the controllers' union -- the Canadian Air Traffic Controllers Association (CATCA) -- appeared to have reached an all-time low. One hundred of the union's 1,800 members were threatened by the government with disciplinary action, including possible dismissal, heavy fines or jail sentences.

The controllers and all other federal public servants in Canada were granted the right to strike in 1968. But Canadians, who have just weathered a 42-day stoppage of mail service by striking postal clerks, have grown increasingly vocal lately about their impatience with this law.

Much of this discontent has been aimed at the air traffic controllers since 1976, when they crippled air service for three days in a national walkout. Public annoyance with CATCA flared once again last fall when the union caused further disruptions of service.

At that time, the federal government went to court to get an injunction against further walkouts by union members, and in April federal authorities unsuccessfully tried, in another court action, to wrest from the union its right to strike.

One Ottawa daily said in an editorial that the controllers' union's refusal Monday to handle U.S. flights was "vigilante action in the name of public safety." It continued: "Transport Minister Jean-Luc Pepin is right to adopt the get-tough stance he has."

The union has been without a contract since Dec. 31, 1980, and negotiations with federal officials over wage increases and improvements in working conditions broke down last spring. The controllers have remained on the job pending a study of their pay and benefits by a consulting firm, but union members could be in a legal position to strike by as early as next month.

This factor was not mentioned by any of the parties to the agreement signed this morning, but government sources maintained privately that union officials were clearly worried that public dissatisfaction with their recent actions might undermine support the union will need if it does go on strike.

Union leaders denied that they had been forced to back down in signing today's agreement, and cited its provisions for a joint union-government committee to investigate flight safety conditions.

But the union's insistence that conditions were unsafe was widely discounted by Canadians.

The union reported 41 dangerous or irregular flight incidents in recent days. A federal official said, however, that the government had investigated 20 of these incidents by Monday night and concluded that none of them was dangerous and that most of them were not even irregular.

Although Pepin spoke to U.S. Transportation Secretary Drew Lewis before announcing tough measures against the Canadian controllers Monday, Canadian officials said Ottawa had not been under any pressure from by the Reagan administration to end the problem quickly.

But other pressures on the Canadian government seemed to have speeded agreement. The International Civil Aviation Organization, which coordinates worldwide air control systems, began moving yesterday to remove transatlantic flights from Canadian control. The group sent a telegram to Pepin saying member countries wanted to discuss contingency plans with Canadian officials this morning.

Officials of Transport Canada said temporary loss of control at Gander, Newfoundland, which handles about 400 transatlantic flights daily, could have led to a permanent loss of controllers' jobs at that center.

"That bothered me quite a bit," Pepin said. "Probably CATCA too, and the people in Gander."