Britain and Ireland are increasing their security forces along the border between Ireland and British Northern Ireland, concentrating more soldiers and police in areas frequented by Provisional Irish Republican Army terrorists.

The British Army and Ulster police also are intensifying their covert intelligence operations in Northern Ireland. The Irish government reportedly is considering using special antiterrorist squads for similar operations south of the border with more sophisticated communications with Ulster's security forces.

Stronger security measures along the Irish-Ulster border have been made necessary by the alarming success of the reorganized IRA's campaign of selective shootings and bomings. Last year 81 people died in sectarian violence in Ulster, compared to a peak during the decade-long British military presence of 467 in 1972. But in just eight months of this year, the IRA has killed 73, mostly in ambushes near the border.

Thirty British soldiers have been killed this year, including 18 who died in the twin-bomb ambush two weeks ago at Warrenpoint in Ulster. Five hours earlier, Lord Mountbatten and three other people were killed when his fishing boat was blown up just off the Irish coast about 10 miles from the Ulster border.

These deaths -- the worst single day's toll since British troops were sent to Ulster almost exactly 10 years earlier -- prompted Britain to speed up its reinforcement of soldiers and Ulster police along the border.

Plans to enlarge the Northern Ireland police force from 6,500 to 7,500 officers were announced a few days after Mountbatten and the 18 soldiers were killed. The number of British soldiers in Ulster remains at 13,000, but the troops are being concentrated more with the police near IRA strongholds along the border.

The military and police also are stepping up their surveillance activities. This includes using informers, observations from systematic raids and visits at homes of suspected IRA members and sympathizers in Ulster, helicopter and remote control television camera surveillance of the border, electronic eavesdropping and other covert methods.

This year, according to Britain's Northern Ireland secretary, Humphrey Atkins, the Army and police in Ulster have charged 380 suspects with terrorism, including 29 charged with murder, and have intercepted or disarmed a ton of explosives, including bombs and electronic detonators.

Ulster security officials want the Irish police and Army to plant antiterrorist squads south of the border to monitor activities of known IRA terrorists operating from towns just inside Ireland. The undercover squads might catch them and seize their explosives and guns, or alert the Ulster security forces when they cross the border.

This is the highest priority on the list of measures British and Ulster officials have urged the Irish government to take to improve security co-operation on the border, Ulster security sources said last week. Controversial proposals for "hot pursuit" by British soldiers or Ulster police of terrorists fleeing across the border is not as important, they said.

The British also have asked the Irish government to improve communications between security forces on both sides of the border. British troops now can communicate with the Irish Army only by going through the Ulster police, who must telephone the Irish police. Ulster security forces would like to use instantaneous electronic communications, particularly in emergencies.

British and Ulster officials would like to work out a way for terrorists who commit crimes in Ulster or the rest of Britain and then return to Ireland to be brought out for questioning, and be charged and tried.

Since 1971, Irish judges have turned down 75 extradition requests for IRA terrorists on the grounds that the Irish constitution prohibits extradition for political crimes. Even after Ireland signs the new European Economic Community extradition treaty, which it is expected to do this autumn, extradition could be blocked under a similar clause in the EEC treaty.

These and other requests for security assistance were made by British Prime Minister Margaret Thatcher after Mountbatten's funeral in a meeting with Irish Prime Minister Jack Lynch, who said he also suggested ways the British might improve security on their side of the border by concentrating more forces there.

Security officials in both governments are studying these suggestions, with final decisions expected to be reached at a meeting of British and Irish Cabinet ministers in about three weeks.

A secret assessment of the Provisional IRA completed by British and Ulster intelligence officials last November, when the current IRA offensive began, stated that Ireland "provides a safe mounting base for cross-border operations and secure training areas. The provisional IRA's logistic support flows through [Ireland], where arms and ammunition are received from overseas. Improvised weapons, bombs and explosives are manufactured there."

"Terrorists can live there without fear of extradition for crimes committed in the north. In short," concluded the report, which gradually has become public since being stolen earlier this year by the IRA, Ireland "provides many of the facilities of the classic safe haven so essential to any successful terrorist movement. And it will probably continue doing so for the foreseeable future."

According to this assessment and Ulster security officials, the Provisional IRA reorganized itself during the past year -- with the help of international terrorist organizations -- from a loose military structure that had been riddled by security forces by the end of 1977 into a leaner, classic revolutionary cadre of no more than 500 men and women in small individual cells.

"They cannot sustain the same high volume of violence as before," said one security official, "but they can plan better and act more effectively, and they are harder to penetrate."

The British security assessment warned of "a strata of intelligent, astute and experienced terrorists who provide the backbone of the organization . . . Our evidence of the caliber of the rank-and-file terrorists does not support the view that they are merely hooligans drawn from the unemployed and unemployable."

"The active service units are for the most part manned by terrorists tempered by up to 10 years of operational experience," the assessment continues. "They are continually learning from mistakes and developing their expertise. We can therefore expect to see increased professionalism and the greater exploitation of modern technology for terrorist purposes."

This already is evident, beginning with the night of March 22, when 49 bombs were planted in 20 cities and towns of Ulster by an estimated 100 terrorists in just 90 minutes. The recent ambush that killed 18 soldiers, and the assassination of Mountbatten, both incidents in which radio-detonated bombs apparently were employed, display the advanced technology the intelligence report had predicted.

The security report estimates that the Provisional IRA raises about $2 million each year from bank robberies in Ireland and Ulster, after-hours clubs and other racketeering, contributions from foreign sources and even from legitimate welfare and prison relief payments.

The money pays for salaries of active terrorists, averaging $50 a week for about 250 of them and $100 a week for about 60 top operatives, plus support for prisoners, publishing propaganda, and buying explosives and weapons.

Guns are smuggled from American and international terrorist sources, and bombs and sophisticated "home-made" mortars are manufactured in clandestine weapons factories, according to security officials. These officials expressed concern that the terrorists might obtain mobile antiaircraft missiles within the next several years.

Adding to the concern of security officials on both sides of the Irish border are several shootings of Catholics in Belfast and along the border a that officials attribute to long-dormant Protestant paramilitary groups seeking revenge for recent Provisional IRA attacks. Such action, officials fear, will only escalate the violence.