Britain will convene a conference of Protestant and Catholic political leaders in Northern Ireland next month to consider British proposals that they share power in an Ulster home-rule government.

The conference was made possible yesterday when John Hume, the new leader of the Catholic Social Democratic and Labor Party, agreed to attend after British officials promised that the conference could discuss the so-called "Irish dimension" -- links between British-ruled Ulster and the adjoining Republic of Ireland.

The British will be seeking from the Ulster politicians a consensus on a form of limited self-government, with sufficient power for the Catholic minority. Complete agreement by all parties at the conference will not be required, as long as the British believe both Ulster communities will support whatever Parliament enacts in London.

Although previous attempts at self-government and power-sharing in Northern Ireland ultimately have failed, Britain and other interested countries -- including the United States -- believe that only through increased political cooperation between Protestants and Catholics can sectarian peace be achieved in Ulster.

Just today, five more British soldiers were killed in two separate mine attacks in Ulster near the border with the Irish Republic. So far this year 38 British soldiers have died in Northern Ireland, and the death toll for all security forces, including prison personnel, is 70.

In all, sectarian violence in Ulster claimed 109 lives this year, as a reorganized Irish Republican Army has stepped up its campaign against British rule.

Hume's agreement to attend the Ulster conference came just in time for British Prime Minister Margaret Thatcher to tell President Carter at the White House Monday that her political initiative in Northern Ireland has been launched successfully. In return, Thatcher will ask Carter to authorize the export of firearms needed by the Ulster police.

Although other British officials were not certain that the time was yet ripe, Thatcher was determined to move now to demonstrate to American critics that she wanted to do more than just crack down on IRA terrorists.

She pushed her Northern Ireland secretary, Humphrey Atkins, to announce plans for the Ulster conference in Parliament here just before she went to dinner at the American Embassy here to discuss her official visit to the United States. The initiative appeared doomed however, when only one of the four Ulster political parties invited to the conference, the nonsectarian Alliance Party, agreed to attend.

Thatcher then made clear that whether they liked it or not, she intended to give Northern Ireland limited self-government and transfer to it some of the almost total governmental responsibility the British government exercises.

Shortly after a meeting with Thatcher, the Rev. Ian Paisley, leader of the Democratic Unionist Party and regarded as Ulster's most militant Protestant leader, agreed to attend the conference.

Gerry Fitt, then leader of the Social Democratic and Labor Party and a member of the British Parliament along with Paisley, also said he would attend. But he was overruled by his party's executive committee because the British discussion document for the conference ruled out any consideration of Ulster-Irish links.

Fitt then resigned as party leader and was replaced by Hume, Ulster's only Catholic member of the European Parliament. He began negotiations with Atkins that won the British promise of discussion of the "Irish dimension."

The only remaining holdout is James Molyneaux, another member of the British Parliament and leader of the official Unionist Party, the chief Protestant rival of Paisley's party in Ulster. Molyneaux is believed to be acting under the influence of Enoch Powell, a former Conservative member of Parliament who switched to the official Unionists in Ulster.

Pressure is growing on Molyneaux from others in his party to change his mind. They fear that they have been outflanked by Paisley, who would be the sole spokesman for Ulster's Protestants at the conference.

Paisley, a fiery Belfast minister with a fiercely loyal following, is believed to see the conference as an opportunity to become the leading Protestant figure in a new Ulster home rule government. He recently has become noticeably more conciliatory toward Ulster Catholics, Ireland and British officials in Uslter.

When Atkins announced in Parliment that he would dallow discussion of Ulster-Irish links at the conference after all, Paisley surprisingly agreed that there was an Irish dimension to be discussed.

British officials see Paisley's behavior as one reason for cautious optimism about the conference. Another is the fact that Paisley and Hume have clearly become the strong leaders of the Protestant and Catholic communities, respectively, in Ulster.

They were the leading vote-getters in the province's proportional voting for the European Parliament, and both are internationally known. Hume, a veteran Catholic leader from Londonderry in western Ulster, is particularly close to Sen. Edward Kennedy (D-Mass.) and House Speaker Thomas P. (Tip) O'Neill (D-Mass.).

"This is only the first hurdle," Hume cautioned yesterday about the agreement to convene the conference. "It is one step at a time. Whether we will reach ultimate agreement is another matter. Nobody is under any illusions about the major differences between the parties."