Irish Prime Minister Charles Haughey has mounted a determined diplomatic effort to dissuade Britain from trying to set up a new Protestant-dominated provincial government in British-ruled Northern Ireland.

Haughey and Irish diplomats and officials are warning that the limited home rule the British government is considering for Ulster is doomed to failure because it offers too little power and protection for the Catholic minortiy and no possibility of Ulster's eventual unification with the rest of Ireland.

A British Cabinet committee headed by Prime Minister Margaret Thatcher today reviewed several home rule possibilities -- ranging from a consultative assembly to advise British officials ruling Northern Ireland to a fullblown provincial legislature and cabinet with authority over almost all local matters except law enforcement.

Each would be controlled by Ulster's two-thirds Portestant majority through direct elections, but complicated weighted voting measures would give the Catholic minority at least veto power over the budget or major decisions.

British officials want to be ready to present these proposals to the British Prliament and to Protestant and Catholic leaders in Northern Ireland later this month for discussion. They hope to find consensus support for one of the plans, enact it and hold elections in Northern Ireland next year.

But Irish officials and Protestant and Catholic political leaders in Northern Ireland do not expect a consensus to be reached. While Dublin and Ulster Catholic leaders consider the minority protections reportedly under consideration to be too weak, Ulster Protestant leaders have said they would reject any attempt to force them to share power with the Catholic minority.

Haughey is trying to persuade Thatcher to abandon this strategy as futile and likely to produce further dangerous political frustration in Northern Ireland. Instead, he wants her to agree that Britain must eventually withdraw from Ulster and help unify it with Ireland.

Hughey has suggested that Thatcher push Ulster Protestant leaders to sit down with British and Irish officials to negotiate a way for Ulster to be joined with predominantly Catholic Ireland in a federal arrangement that would protect the Protestant' interests.

Haughey is offering to make drastic changes in the Irish constitution to allow Ulster to retain many of its own laws, including more lenient legislation than Ireland on divorce, contraception and "other things about which the Protestants have strong feeling."

In television interview broadcast in Britain last night, Haughey said, "it's quite possible to have complete sets of social legislative provisions in one area from the other.

"If the Ulster Protestants were prepared to come to a conference table, to sit down with us to discuss some new arrangement, all of that can be discussed." Haughey said.

Haughey said Britain's eventual withdrawal from Northern Ireland "ultimately will come to be recognized as being in the best interests of both the British people and the Irish people." Ireland can never be a stable, close friend and ally of Britain, Haughey said, "as long as there is a British military and political presence in Ireland."

Haughey's arguments are being reinforced by Irish diplomats who warn privately that British pursuit of another attempted "internal solution" to the sectarian problem in Northern Ureland can only set back British-Irish relations. If the new British initiative fails, they add, it once again will leave a vacuum to be filled by the still active and dangerous Provisional Irish Republican Army terrorists.