The outcome of Ireland's extremely close election remained in doubt early today in a dramatic vote count highlighted by surprising support in strongly nationalist regions of the country for nine Irish nationalist terrorists imprisoned in neighboring Northern Ireland.

In the tree multiseat constituencies along the border with British-ruled Ulster, one of the prisoners won a seat in the Irish parliament. Another was on the verge of winning a seat, and a third fell just short. The prisoners' showing greatly exceeded what was expected by politicians and analysts here, and could affect whether any party or coalition is able to win a working majority to tackle the country's economic problems and the question of Northern Ireland.

The ruling Fianna Fail Party of Prime Minister Charles Haughey led in the slow-moving count of Thursday's balloting under Ireland's complicated system of proportional representation. But, as expected, its share of the votre dropped from its 1977 landslide victory and its majority in the expanded 155-seat parliament appeared in jeopardy.

The biggest opposition party, Fine Gael, led by economist and former foreign minister, Garrett Fitzgerald, made big gains with its proposals for stimulating the economy, including a sizable income tax cut and a cash tax rebate for housewives. But the union-based Labor Party, Fine Gael's prospective partner in any coalition government to replace Fianna Fail, suffered a significant loss of support, with its leader, Frank Cluskey, losing his seat in parliament.

Leading politicians and analysts speculated that the country might be left with a minority government that would have difficulty making major changes in economic policy to cope with high inflation, increasing unemployment and the government's burgeoning foreign debt. This also could affect talks begun by Haughey and British Prime Minister Margaret Thatcher on the relationship of Ireland, Britain and Ulster.

The major surprise in the election has been the strong showing of the nine convicted Irish nationalist terrorists in British prisons in Northern Ireland who were put up as candidates for the Irish parliament, taking advantage of cross-border citizenship rights.

One of the prisoners, Patrick Agnew, easily won one of the four seats from Lough, a county on the border with Northern Ireland where Agnew's home town of Dundalk is located. In the Canan-Monaghan constituency on the border, prisoner Kiernan Doherty, of Belfast in Northern Ireland, was near the 20 percent of the vote needed to win one of the five seats there.

Joseph McDonnell, one of five inmates of the Maze Prison in Belfast currently on hunger strike to dramatize demands for prisoner-of-war status for convicted Irish Republican terrorists in Northern Ireland, was narrowly eliminated from contention for a seat from Sligo-Leitrim. This constituency is just across the border from the one in Northern Ireland where hunger striker Bobby Sands won the election to the British Parliament four weeks before his death May 4.

McDonnell, who replaced Sands on hunger strike, is the nearest to death among the five now refusing food. After 35 days without eating, he was reported this week by hunger strike supporters to have lost 32 pounds and to be feeling weak.

With three other hunger strikers and the rest of the prisoners standing in the election winning an average of about 10 percent of the vote in their constituencies, they appeared to have competed for support primarily with Fianna Fail, the Irish political party most closely identified with aspirations for a unified Ireland.

Irish Foreign Minister and Fianna Fail candidate Brian Lenihan said tonight that there was "a highly emotive content" in the vote for the prisoners. "Given our past history," Lenihan said, "there are still some people who will vote emotionally on that issue."

"I think it's a vote of sympathy for the minority community in Northern Ireland," said Joseph McCartin, a Fine Gael candidate who won election against hunger striker McDonnell in Sligo-Leitrim.

Bernadette Devlin McAliskey, a former member of the British Parliament from Northern Ireland who is a leader of supporters of the hunger strike, said she was not surprised by what she called "a groundswell of support [in the Irish Republic] for the prisoners and a demand for action and not just words" by the Irish government. The results show, she said, that voters in Ireland "are more sympathetic to the prisoners than the government has been."

Haughey, who was harassed by hunger-strike supporters throughout the election campaign, has asked the European Commission on Human Rights to find a compromise solution to the prisoners' demands for changes in the prison regime in Northern Ireland and Irish officials privately have urged the British government to be more flexible.

But Haughey has avoided directly criticizing the British government or Thatcher. He repeated during the campaign his belief that the current dialogue eventually could lead to a solution of the impasse in Northern Ireland, where the two-thirds Portestant majority wants to remain part of Britain, while the one-third Catholic minority seeks unity with Ireland.

[With 32 of 41 constituencies counted, representing 117 seats, Fianna Fail had 56 seats, Fine Gael won 50, Labor claimed 7 and independents 4, representing a swing of 5.8 percent of Fine Gael and a loss of 5.3 percent by Fianna Fail and 1.4 percent by Labor, United Press International reported. The Irish nationalist convicts were garnering 2 percent of the ballots nationwide.]