Charles Haughey, a millionaire Irish businessman and charismatic politician best known outside the country for his strong nationalism, was chosen today to be Ireland's new prime minister.

He won the leadership of the ruling Fianna Fail Party in a close contest with the deputy premier and finance minister, George Colley. The party's members in the Irish parliament in Dublin gave Haughey 44 votes to 38 for Colley.

Haughey, 54, who had been health minister, will be formally elected prime minister by parliament next week when the present prime minister, Jack Lynch, officially resigns. Lynch touched off a spirited two-day lobbying campaign inside Fianna Fail when he suddenly announced his resignation Wednesday.

Asked at a press conference today about his support for the unification of Ireland with British-ruled Northern Ireland, Haughey pledged to seek that goal only through peaceful means. He condemned the terrorist campaign of the Provisional Irish Republican Army.

He said he would continue Irish cooperation with the British on antiterrorist measures along the Irish-Ulster border and would generally follow Lynch's moderate approach toward the Ulster problem.

"There may perhaps be some differences of emphasis on particular aspects of policy," Haughey added, however. "I regard the peaceful unification of Ireland as my primary priority."

Lynch recently ran into strong criticism from inside Fianna Fail (Soldiers of Destiny), a traditionally nationalist party, by telling the British he would not insist that a formal assurance of Irish unification be included in an embryonic British plan for limited home rule in Northern Ireland, in which the Protestant majority and Catholic minority could share power. One of Lynch's most outspoken critics was young Sile de Valera, granddaughter of the late Irish patriot, president and prime minister, Eamon de Valera. Sile de Valera also is one of Haughey's strongest Fianna Fail supporters.

This criticism, along with a slowdown in Ireland's recently rapid economic growth and Fianna Fail's poor showings in recent elections, forced Lynch to resign months ahead of his intended retirement after serving as prime minister for nine years.

Haughey, a self-made millionaire who lives on a 280-acre estate north of Dublin, married the daughter of Sean Lemass, Lynch's predecessor as Fianna Fail leader and prime minister. A member of the Irish parliament since 1957, Haughey first became a Cabinet minister in 1961. Later, as finance minister, he made Ireland a tax haven for writers and artists.

Haughey was forced to resign as finance minister in 1970, when he was accused of conspiring in an illegal scheme to import arms for the IRA. He was acquitted of the charge and has vigorously maintained his innocence. Today, he called the court case "a matter of history."

Although the British media have speculated that Haughey's nationalism could threaten British-Irish relations and cooperation on Ulster, sources here said British government officials are taking a cautious approach and will judge him only on his performance as prime minister.

Knowledgeable sources in Dublin said Haughey is likely to be preoccupied with refueling the Irish economy, reviving Fianna Fail's political strength, and seeking to establish himself within the European Economic Community, a key to Ireland's economic fortunes.

They said Haughey had admired the pragmatism of his father-in-law, Lemass, who made the first contacts as Irish prime minister with leaders in Ulster. If the British government could make its new political initiative in Ulster acceptable to both Protestants and Catholics there, according to one observer, Haughey would be able to help make it work because of "his impeccable republican credentials." CAPTION: Picture, CHARLES HAUGHEY . . . pledges a peaceful quest of unity