Results in the election for members of a new Northern Ireland assembly have shown unexpected strength for the Irish Republican Army's political wing, the Sinn Fein, and appear to assure that the assembly will fail as a workable forum.

About a fourth of the vote in the Wednesday election went to representatives of Roman Catholic parties, including Sinn Fein, that are pledged to boycott the assembly. The largest number of seats were won by the Protestant, pro-British Unionist Parties. But without what is called "cross-community" backing, the assembly's deliberations are expected to have little effect.

The assembly idea was put forth by James Prior, secretary of state for Northern Ireland in Prime Minister Margaret Thatcher's Conservative government. It was Britain's sixth major attempt in a decade to establish a means for political dialogue and local authority in Northern Ireland.

Sinn Fein and the main Catholic moderate party, the Social Democratic and Labor Party, had said from the outset that they would participate in the election but refuse to take their seats as a protest against Britain's role in Northern Ireland. The only hope for a political breakthrough was that a substantial number of Catholic voters would reject that view by choosing candidates committed to giving the assembly a chance.

Instead, at least five Sinn Fein members have been elected -- a surprisingly strong showing in the view of many politicians. Sinn Fein, which has refused to take part in past similar elections, won about 10 percent of the vote and about 40 percent in Catholic areas where its candidates faced the Social Democratic and Labor Party.

The nonsectarian, moderate Alliance Party won at least eight seats and along with the Protestant Unionist Parties will comprise the assembly. Under Prior's plan the body is to have consultative and debating rights over the British government's direct rule of the province, but no responsibility. The hope had been that in time, the group might attract sufficient community backing -- 70 percent was Prior's figure -- for it to be given legislative authority to run the province.

Prior acknowledged that the results were a setback. But he said that "everything in Northern Ireland is a setback. Unless one is expecting miracles, it is difficult to ask people who have held contrasting views for 300 years to work together."

He said sectarian differences dating back for centuries would not be easily overcome but he said the assembly would go "patiently on."

Prior said he would ask the party leaders to select a date for the assembly to convene, but refused to say whether he would include Sinn Fein in the process.

A former Labor Party secretary of state for Northern Ireland, Merlyn Rees, said at least two of the five winning Sinn Fein candidates had been "very involved in murder and killing. . . . Now they have been elected and are far more powerful than they were when they only held a gun."

Gerry Adams, a party vice president who spent 4 1/2 years in prison for suspected terrorism, said tonight that he would sue Rees for slander.

The election shows the continuing polarization of Northern Ireland's community rather than any significant new trend toward compromise. As many in Northern Ireland anticipated that outcome, there was considerable sentiment there earlier this week that the effort was misguided.

"Why set up us and himself for failure?" one long-time political analyst in Belfast asked, referring to Prior's plan. "We'd be better off leaving things alone."

The campaign produced a spate of violence, generally linked to the Irish National Liberation Army, an IRA splinter, that claimed the death of eight people. "What the contest accomplished," the same analyst said, "is to give greater political credibility to the IRA. That is not going to bring a solution in Northern Ireland any closer."

Among others elected to the assembly were the two main Protestant leaders, James Molyneaux, whose Official Unionist Party won the largest share of the seats, and the Rev. Ian Paisley of the more militant Democratic Unionist Party.

With 68 of the 78 seats decided, the Official Unionists had 24, the Democratic Unionists 19, the Social Democratic and Labor Party 10, the Alliance Party 8, Sinn Fein 5 and two Protestant splinter parties one each.