The factory worker wore a button on her lapel where she used to attach a Solidarity union pin, in plain sight of whoever cared to notice. Because Solidarity is now banned, she had instead a picture of the Black Madonna, Poland's holiest icon.

To her both buttons meant the same thing: independence from Communist institutions, belief in a credible authority, freedom to think and to act.

This woman on the factory floor of the Olympia textile works here also made no secret of why she and most of the other 3,200 plant employes were refusing to join the new official trade unions. "We don't trust them," she said plainly.

The refrain is heard throughout Poland. Four months after the dissolution of the independent trade union Solidarity, two months after the suspension of martial law and the start of registration of the new unions, Polish workers continue to regard the revised structures with deep distrust.

By the latest official count, 1.25 million people--or about one eighth the number claimed by Solidarity at its peak--have joined the new unions. That is just a little more than half the membership of Poland's Communist Party elite, although there are indications that party members may be refraining from joining the unions en masse to avoid making the new groups look like party instruments--which is what many workers suspect they really are.

The absence in Poland of a new union movement is frustrating government efforts to restore at least the appearance of a resumed dialogue with workers. At the same time, officials say they are aware that pressing workers to join the unions would risk discrediting the new organizations altogether.

"We are still in the initial phase, we can't rush the process," Tadeusz Czechowicz, a member of Poland's ruling Politburo, told a visiting group of foreign correspondents in Lodz last week. "Workers recently have gone through psychological shocks and ideological uncertainties." He suggested that many Poles are shying from the new unions out of a sense of exhaustion and confusion rather than resentment against the authorities.

The new trade union law, adopted in October 1982, sets stringent conditions for resumption of union activity. The conditions are intended to prevent any future Polish union from attaining the national political power that Solidarity wielded.

In contrast to Solidarity, which owed much of its power to a strong regional structure, the new unions are divided along individual craft lines. Only one union is allowed to operate in each factory until 1985, when more might be permitted. The formation of a national federation of unions also is delayed until then.

This lack of an association of unions poses an immediate dilemma for the government in its efforts to appear to be consulting the new unions on economic decisions. More than 5,600 unions already have been registered by the courts and 60 to 100 more union applications are being received daily, too many groups for the government to deal with individually.

Stanislaw Ciosek, the minister for trade union affairs, told to foreign reporters last week that the Polish government currently is stumped on how to join the unions into consultative units that would allow officials to canvass worker opinion. "There are consultations now on how to carry out consultations," Ciosek said, revealing the confusion at the top.

In a speech Monday to workers in Katowice, Polish leader Wojciech Jaruzelski announced the government would be speeding up the process of forming multiunion associations "wherever the conditions will arise and wherever trade union representatives will be elected." He added that a resolution was being prepared as well "to expand the forms and possibilities" for trade unions to voice opinions on legislation affecting workers' interests.

Leaders of the Solidarity underground, meanwhile, continue to urge a boycott of the new unions. They have called on workers to set up social aid committees independent of the unions and to insist on nonunion holiday and credit funds in their factories.

In addition, former Solidarity activists have objected to the transfer of Solidarity's extensive assets to the new unions. The old union's property, they say, should be frozen for several years until the time the law is supposed to permit formation of more than one union in individual factories.

Ciosek acknowledged that the boycott had been effective, particularly in many of Poland's larger enterprises, reinforcing what the minister termed the "natural reserve" many workers feel these days toward unions generally. Suggesting also that some dirty play is involved, other Polish officials have complained of harassment, including threats of physical violence, against workers inclined to organize or join the new unions.

Underground newsletters, in turn, have accused the authorities of offering special favors to employes who will set up or become members of the new unions.

Actually, what may be more significant than the relationship between the unions and authorities is how the unions develop in relation to new workers' councils. These factory-level councils, a central element in Poland's economic reform written into law before the December 1981 military crackdown, are to be freely elected bodies with responsibility for setting targets and policies--including wage scales--in individual enterprises.

They are forming slowly in the face of reluctance by state and party bureaucrats to cede control, but the power that is supposed to be vested in them, together with their democratic basis, make the councils of keen interest to some former Solidarity supporters.

"We're looking forward to the council elections," said the woman with the button, an ex-Solidarity activist. "We expect to be active there."