Britain and Ireland took a major step today toward improving relations with each other and reconciling the Protestants and Catholics of violence-torn Northern Ireland through the formation of an Anglo-Irish intergovernmental council.

British Prime Minister Margaret Thatcher and her Irish counterpart, Garret FitzGerald, agreed at a five-hour meeting here to set up the council, at which ministers and top officials would deal with issues affecting British-ruled Northern Ireland and the Irish Republic.

But the two leaders delayed plans for formation of a parliamentary council, which would include politicians from London, Dublin and Belfast, the capital of Northern Ireland.

This council of politicians also would have discussed ways of ending violence between Catholic extremists trying to unite Ireland by force and Protestant extremists determined to keep it united to Britain.

The two leaders decided to wait, partly to see how the government council works and partly because of bitter opposition from Protestant northerners, led by the Rev. Ian Paisley, who oppose any kind of reconciliation with the province's minority Catholic population.

They said a parliamentary council would have to be set up by the British and Irish parliaments and by a Northern Irish assembly if one were formed.

In an obvious bid to try to create some kind of formal contact between the communities, they agreed to try to set up an advisory committee associated with the intergovernmental council "on economic, social and cultural cooperation, with a wide membership."

It was felt that such a committee, with a wide-ranging agenda and with representative figures from the Catholic and Protestant communities, could become the nucleus for a communal dialogue.

Today's agreement was seen as a symbol of closer cooperation between Britain and Ireland, whose relations often have been strained in the past, particularly during the recently ended hunger strikes of Irish nationalist prisoners in Northern Irish jails.

"Our meeting was warm, friendly, practical and constructive and indicated steady progress forward," Thatcher said.

President Reagan, speaking in New York last night at the 84th annual American-Irish Historical Society dinner, made no mention of the Thatcher-FitzGerald agreement, but said: "I think we all should pray that responsible leaders on both sides, the governments of the United Kingdom and the Republic of Ireland, can bring peace to that beautiful isle once again."

FitzGerald, who has been a vigorous promoter of better relations with the British government and Northern Ireland, told reporters after the meeting at 10 Downing Street: "Each concrete step taken brings us closer to solving the problem, although clearly there are many more steps to be taken."

In a statement after their meeting, the two prime ministers said they wanted to "promote arrangements which might help to reduce tensions between, and to reconcile, the people of the two parts of Ireland."

The one significant difference between them was over the unification of Ireland, long a goal of the Catholic community and bitterly opposed by the Protestants, who would be in a minority if Northern Ireland were joined to the Irish Republic.

FitzGerald declared that it was the wish of his government and "the great majority of the people of the island of Ireland to secure the unity of Ireland by agreement and in peace."

But he agreed with Thatcher that it could be done only with the consent of the province's Protestant majority, a point the British premier repeatedly stressed.

In a hint that she might push for a referendum in Northern Ireland, Thatcher declared that if the Northern Irish voted for unity, the London government would accept their decision.

In a second move aimed at improving relations, the two prime ministers agreed to increase cross-border economic cooperation, particularly on electricity and natural gas. The south has inexpensive gas and the north has surplus electricity.

The electricity link between north and south has been blown up repeatedly by the Irish Republican Army (IRA) at a cost of about $500,000 each time, and Thatcher said efforts were being made to work out a way of reopening the connection, possibly across the Irish Sea instead of the vulnerable land border.

Today's meeting was the second round of British-Irish talks initiated last year by FitzGerald's predecessor, Charles Haughey, and was attended by the British foreign secretary, Lord Carrington, and his Irish counterpart, James Dooge, as well as other senior Cabinet ministers.

Despite Thatcher's insistence that no changes would be made in Northern Ireland without popular consent, there was a chorus of protests from Protestant leaders there to today's developments.

Paisley turned up at 10 Downing Street 45 minutes before FitzGerald to present a letter protesting that the meeting was taking place. An Anglo-Irish council would, he said, "meet with united, relentless and unqualified opposition from the Protestants of Northern Ireland."

Paisley accused Thatcher of negotiating the province's future with FitzGerald and said that the meeting was causing heart-searching and concern.

The leader of the Orange Order of the Ulster Unionist Party, which insists on union with Britain, said "there can be no private deal because the northern unionists have their own view."

While the Northern Irish Protestants stand ready to block any moves toward reconciliation, FitzGerald is under pressure to produce results. He leads a shaky coalition and may call an election within a year.