Both Protestant and Catholic political leaders in British-ruled Northern Ireland have indicated they are ready to discuss the British government's new proposals for home rule in Ulster, although they have strong reservations about some of them.

After Prime Minister Margaret Thatcher's government revealed its proposals yesterday, Ulster's most popular and militant Protestant leaders, the Rev. Ian Paisley, who heads the Democratic Unionist Party, said, "Quite a bit of this document is constructive."

Paisley said he considered the British proposal for an Ulster Cabinet in which the Protestant majority and Catholic minority would share power to be "totally unworkable." However, he saw possibilities in a proposal to give the minority some machanism in a new Northern Ireland legislature to appeal, if not block, majority decisions.

Paisley, who led the Protestant opposition that wrecked an attempt to set up a power-sharing government in Ulster in 1974, has been much more conciliatory in recent months. British officials believe that he may be ready to agree to a compromise Ulster home rule plan that would enable him, as Northern Ireland's leading vote-getter, to become its chief executive.

James Molyneaux, leader of the rival Protestant party in Northern Ireland, the Official Unionists, was more negative in his reaction to Thatcher's political initiative.

But after boycotting last winter's preliminary consulations with Thatcher's Northern Ireland secretary, Humphrey Atkins, in which Paisley participated, Molyneaux indicated he now was ready to discuss the new proposals with Atkins. Despite his reservations, Molyneaux said, the proposals had created "a different ball game."

The Irish Republican Army, which has been fighting to end British control of Northern Ireland and unite it with the predominantly Catholic Irish Republic, rejected the British proposals.

The most important Catholic leader in Northern Ireland, John Hume of the Social Democratic and Labor Party, said he saw much in the document that could be the basis for negotations. Now, however, he must convince more militant members of his party who have insisted that Britain declare its willingness to allow the eventual unification of Ireland and Ulster. Northern Ireland remained part of Britain when Ireland was granted independence in 1920.

Hume said, however, he was pleased that the British white paper stressed that any new governmental structure in Northern Ireland must be acceptable to both Catholics and Protestants. "I also note with interest the reference to the need . . . to develop what is described as the unique relationship between the peoples of Northern Ireland, Britain and the republic," Hume said.

The Irish government also was pleased by this and by the white paper's encouragement of the development of existing links between Ulster and Ireland in the fields of economics, energy, culture.

"This is movement in the right direction," said one Irish government source, "although not as far as we wish."

Irish officials still are doubtful that Protestants and Catholic leaders in Ulster can agree on a home-rule plan that would provide Catholics with enough protection and power to satisfy them without making the plan unacceptable to the Protestants. The officials fear that failure would only increase political frustration and sectarian strife in Northern Ireland during a period of economic decline.

In the end, Irish officials believe, the Thatcher government will have to take "the more radical step" of involving the Irish government in negotiations leading to eventual unification of Ireland and Ulster. They have hinted that predominantly Protestant Ulster could be given separate legal status and possibly its own regional government within predominantly Catholic Ireland.