Prime Ministers Margaret Thatcher of Britain and Jack Lyncy of Ireland agreed today to "substantially improve" their security forces' cooperation in combating Irish Republican Army terrorism.

In five hours of talks here after both attended the Westminster Abbey funeral of Lord Mountbatten, who was assassinated by the IRA last week, Thatcher and Lynch agreed that their governments would examine a number of each other's suggestions to improve security and would produce final proposals at a meeting of their Cabinet ministers in three weeks.

"There likely will be decisions made" at that meeting on new joint security measures, Lynch said at a press conference tonight after his working lunch and afternoon meeting with Thatcher. "No specific proposals have been ruled out on either side."

This represented some movement by Lynch from his public statements. Earlier this week he appeared to rule out a number of security suggestions that Thatcher was expected to make.

For her part, Thatcher did not press two changes sought by British security forces in Northern Ireland but which Lynch politically is unable to make: Allowing "hot" pursuit of terrorist suspects by Ulster security forces across the border into Ireland and changing Ireland's extradition policy to allow terrorists living or caught in Ireland to be tried in Britain.

According to the official joint statement of the meeting, Thatcher also agreed to "make moves at the appropriate time" to "seek an acceptable way of restoring to the people of Northern Ireland more control over their own affairs."

Lynch, who is pushing for a new British political initiative in Northern Ireland, made clear that any new "form of administration" there must be acceptable to both the Catholic minority and the Protestant majority.He evidently envisions a form of power sharing that could be the beginning of the end of emergency, direct British rule there.

Lynch said he and Thatcher agreed to keep secret the specifics of each country's suggestions for improved cooperation on security. Thatcher did not speak to the press at all.

However, several likely possibilities have emerged:

More troops and police patrols on both sides of the Northern Ireland-Ireland border, particularly in Northern Ireland, where British security forces have abandoned land patrols in some places and perform only helicopter surveillance.

Improved communication between the British and Irish armies, which back up the Northern Irish police and must communicate through the police forces in a cumbersome, undependable process, even in emergencies.

Increased contact between Ulster and Irish police when terrorist suspects are arrested in Ireland that might include more travel into Ireland by Ulster investigators.

Changes in Irish criminal laws to make it more difficult to release suspects on bail and more likely that those convicted for crimes, including terrorists, would receive longer sentences.

At his press conference, Lynch also said Ireland will sign the European Community's new extradition agreement, most likely at the end of November, when Common Market leaders meet in Dublin. Ireland's Common Market presidency expires then.

The agreement is worded vaguely, however, in practice allowing Irish courts to decide whether extradition should be blocked in some cases.

Instead, Ireland's Criminal Law Jurisdiction Act of 1976 specifically provides for prosecution in Ireland of residents accused of terrorist offenses in Ulster or mainland Britain. Ulster and British security forces never have been able to trigger such a prosecution, however, because of the difficulty of assembling evidence and producing witnesses on one side of the border for use in court on the other side.

This problem might be alleviated somewhat by more extensive contacts between the security forces on each side of the border, which appear to have increased during the investigation of Lord Mountbatten's murder and the bombings the same day that killed 18 British soldiers in Ulster. Most of those soldiers were buried yesterday and today with full military honors in their home towns throughout Britain.

Hinting at the possibility of stepped-up patrols on both sides of the border, Lynch told reporters, "There are two sides to the border, so security must be two ways. It is incumbent on the British to have as much surveillance on their side of border as we have on ours. We believe our commitment is greater in some places. There are places where there is no British commitment."

He also pointed out that the Irish government is already exploring ways to toughen its criminal laws to better fight terrorism and curb a crime wave in Ireland. However, he said, any changes in criminal law or procedure "have to be politically and legally possible."