The House of Commons decisively rejected a return to capital punishment in Britain tonight, turning back a determined effort to restore hanging in cases of terrorism.

On a day in which four British militiamen were killed by a terrorist bomb in Northern Ireland, the House voted heavily against bringing back the death penalty, which was abolished in 1965. Members were not required to follow party lines in the vote because it was regarded as a question of conscience. The result reflected sentiment against hanging across the political spectrum.

Prime Minister Margaret Thatcher is a supporter of capital punishment. But her views were rejected by margins of as much as 170 votes in a series of ballots at the end of a day-long debate. Because of the recent Conservative landslide victory that brought dozens of new Tories into Parliament, the results had been expected to be much closer.

However, the majority of members accepted the argument put by such venerable parliamentary figures as former Conservative prime minister Edward Heath that the restoration of capital punishment would do nothing to enhance "the safety of the people of the realm."

The death penalty has remained a heatedly contested political issue in Britain since its abolition--except for the offenses of treason, violent piracy and arson in a Navy dockyard under which it has not been invoked. There have been votes on the subject every time a new Parliament convenes. National opinion surveys show a substantial majority of those polled in favor of hanging, but members of Parliament consistently have defied these popular sentiments.

The terrorism issue dominated the debate. The overwhelming majority of murder cases in Britain are related to the sectarian violence in Northern Ireland. Roy Jenkins, former leader of the Social Democrats, noted that the rate of homicide in Ulster is 600 times as great as in the rest of Britain.

That fact was highlighted today by the death of the four militiamen, members of the predominantly Protestant Ulster Defense Regiment, who died when their vehicle hit a 500-pound land mine on a main road in County Tyrone. The Irish Republican Army later announced that it had been responsible for the mining. Later today two Catholics were found dead in Armagh, and police said the two men were believed to have been executed by the IRA for being informers. The six deaths today represented the worst day of violence in Ulster this year.

James Prior, the government minister responsible for Northern Ireland affairs, was one of those most opposed to restoring hanging for terrorist offenses. He said in a letter released earlier this week that the IRA would consider any executed members martyrs and would seek to benefit from the use of capital punishment. "They are so committed to their cause that they are prepared to risk their own lives as well as destroy the lives of others," Prior wrote. His views carried considerable weight with members of Parliament.

Home Affairs Minister Leon Brittan argued in favor of hanging for terrorist offenses--reflecting the split in Thatcher's Cabinet on the emotional issue. Only a few days ago supporters of hanging predicted that they would win with a majority of as much as 20 votes, but the final tally showed a majority of 116 against hanging in terrorism cases.

The closest vote was on reviving the death penalty for the murder of policemen. Yet even in that vote the margin against capital punishment was 81 votes. After considering restoration of hanging in specific cases, the House voted 368 to 223 against any restoration of the death penalty.

Opponents of capital punishment now are likely to argue that no further reconsideration of the question should be necessary for the foreseeable future. They will say that one of the largest Conservative parliamentary majorities in modern history has shown no inclination to revive the use of execution as a deterrent to crime, and it is hard to imagine what might influence a change of attitude.

Heath, whose remarks were regarded as the most eloquent in the debate, said he had listened to repeated parliamentary considerations of the death penalty question. These debates "over 30 years," he said "have prevented us from giving real attention and resources to the problems of crime."

Although violent crime in Britain has increased substantially since the death penalty was abolished, the rate of such crime remains far below that of the United States and many other western countries.

Jenkins, who served as a Labor Party home affairs minister, joined Heath in the vehemence of his attack on the death penalty. "The finality of the punishment is too great for the frailty of human judgment," he said.

One argument against using capital punishment in terrorism cases is that such trials in Northern Ireland are no longer conducted with juries. Judges in these cases act alone in order to eliminate the possibility of intimidation or violence against citizens sitting on juries.

Speaking in support of the restoration motion, Teddy Taylor, a Conservative, noted that Britain was prepared to spend "tens of millions of pounds for deterrent weapons . . . and should have a deterrent against violent crime."