The U.S. Postal Service refused yesterday to open scheduled contract talks with unions representing most of its 578,000 organized workers, effectively delaying the start of the year's largest labor-management contest.

Leaders of the American Postal Workers Union and the National Association of Letter Carriers, the two largest of the four postal unions, waited 25 minutes for the government's negotiators to come to the bargaining table in a ballroom of Washington's L'Enfant Plaza Hotel.

The USPS team didn't show, but the absence was expected, because of a petition filed with the National Labor Relations Board last week by Postmaster General William F. Bolger, seeking to determine the "appropriate" bargaining structure under which negotiations should be conducted.

Bolger argued in his petition that the two smaller units, the Mail Handlers Union and National Rural Letter Carriers Association, were fighting with the two larger groups and were seeking individual contracts with the postal service. Bargaining under such circumstances would prove "unworkable," Bolger said.

In a letter hand-delivered yesterday to the presidents of the two larger unions, Bolger said: "In light of these developments, it is apparent that negotiations for a new contract to replace the one expiring on July 20, 1981, cannot begin at this time." He said postal service leaders "are as interested as you are to ensure that our employes are not prejudiced by the delay in negotiations that may unavoidably result from the board proceeding."

Bolger also said in his letter that he and his aides "would be available anytime after May 4" to dicuss the possibility of extending the current three-year agreement.

However, APWU President Morris (Moe) Biller and NALC President Vincent Sombrotto dismissed Bolger's actions and words as "deceitful" and announced that they had filed an NLRB petition aimed at throwing out Bolger's case as soon as possible.

The union petition argued that the postal service filing "is without basis in law or fact."

The "two largest of the four nationally recognized unions are seeking to bargain jointly with the Postal Service to reach a new national agreement," the union petition said.

"The two smaller unions are also seeking to coordinate their bargaining, but to conduct it separately from the two larger unions. The only change since 1978 is that the Mail Handlers Union has chosen to negotiate with the Rural Letter Carriers, rather than with the joint bargaining committee established by the APWU and NALC."

"We're really not doing anything different than what was done" in the 1978 contract, Sombrotto said. "Bolger knows that," he said.