The death of hunger striker Joe McDonnell in the Maze Prison outside Belfast early today touched off new street violence, in which a 16-year-old boy was killed, and set back efforts to mediate an end to the protest fast by Irish nationalist prisoners in British-ruled Northern Ireland.

McDonnell, a 30-year-old convicted Provisional Irish Republican Army terrorist, died after refusing food for 61 days. Rioting that followed in the Catholic Irish nationalist inner-city neighborhoods of Belfast and in Londonderry was initially much smaller in scale than the weeks of violent disturbances following the deaths of the first four hunger strikers in May.But vehicles were hijacked and burned, police reported sniper attacks and there were widespread fears of intensified violence in weeks to come.

McDonnell's death came as unofficial mediators from an Irish church group failed in their race against time to win final acceptance by the British government and the protesting prisoners of a proposed settlement of the four-month-old series of hunger strikes.

The voluntary mediators from the Irish Commission for Justice and Peace of the Irish Roman Catholic Church today blamed the government of Prime Minister Margaret Thatcher for stalling movement toward a settlement by repeatedly delaying a promised visit by a British offical to the Maze. The official was to explain to the prisoners goverment proposals for changes in their regime if the fasts ended.

A statement of the government's position finally read to the prisoners several hours after McDonnell's death this morning "was not a serious attempt to find a solution," the members of the commission said, because it fell far short of what they expected from marathon meetings with government officials during the past week. e

British officials insisted that today's statement by Britain's Norhtern Ireland secretary, Humphrey Atkins, could still form the basis for a settlement because it makes clear the government is ready to consider many "real improvements" in prison conditions in Northern Ireland, but only after the hunger strike is ended.

In the Catholic neighborhoods of Belfast, Northern Ireland's largest city, which had been quiet in recent nights as efforts to end the hunger strike reached a climax, McDonnell's death produced familiar scenes this morning of ritual whistle-blowing and trash can lid-banging to bring residents onto the streets.

In the rioting that followed, a policeman and two soldiers were injured by a homemade grenade and two civilians were seriously wounded by plastic bullets. Sixteen-year-old John Dempsey was shot and killed by a soldier when troops fired on a gang of hooded men attacking a bus depot, an Army spokesman said.

Fears were voiced of worse to come if the other seven current hunger strikers die during the next two months when tensions normally run high in Northern Ireland.

Protestant British loyalists traditionally march through the streets on July 12, the anniversary of the Battle of the Boyne won by Protestant King William of Orange, who defeated Catholic King James II. Demonstrations by Catholic Irish nationalists usually mark the anniversay on Aug. 9 of the beginning of internment without trial of suspected terrorists by by the British government during the worst years of violence in the province in the early 1970s. Internment was later abandoned.

Hopes had risen in recent days that a new wave of hunger strike deaths and street violence could be avoided by the intervention of the Irish church commission. It was quietly backed by Irish government contacts with British officials, including direct contact between Thatcher and the new Irish prime minister, Garret FitzGerald.

The commission member worked out a prepared settlement in meetings with British officials, the hunger strikers their relatives and supporter outside the prison, and the paramilitary leader of more than 400 convicted Irish nationalist terrorists inside the Maze. The settlement was based on changes in prison conditions discussed with British officals that the mediators believed should be accepted by the inmates, despite serious initial splits among the hunger strikers, their relatives and militant Irish nationalist leaders outside the prison over compromising on the prisoners' original demands.

The commission had intended to make the proposals public yesterday, when it expected a British official to go to the prison to explain them, which the prisoners had demanded as a guarantee of the government's good faith. But Auxiliary Bishop of Dublin Dermot O'Mahony, head of the fiveman commission team, said in Belfast today that it appeared the commission had been "misled" by the government when the prison visit was repeatedly postponed and Atkins' statement fell far short of what the commission expected

"We believe that by clawing back on the part of the British government, a great opportunity to find a resolution was lost," said O'Mahony.

Atkin's statement said the goverment would do nothing "under duress" of the hunger strike and only alluded to changes that could be made on the contentious issues of prisoners' clothing, work assignments and association with other prisoners.

British officials, noting that the government could not appear to be breaking its principle of not negotiating with the prisoners during the hunger strike, said the statement still leaves plenty of room for most of the changes in prison conditions proposed by the commission members.

Spokesmen for the outlawed Provisional Irish Republican Army rejected Atkins statement and accused the British government of "creating the illusion of moving toward a settlement" to silence critics.

Irish Prime Minister FitzGerald called on both sides to be more flexible, saying the "greatest reponsiblity must as always, rest on those with the greatest power. We have been in touch with the British authorities on many occasions to impress on them the need for an urgent solution." He said McDonnell's death should not be used by anyone "as an occasion to delay the finding of a solution."

John Hume, leader of Northern Ireland's moderate Catholic Social Democratic and Labor Party, criticized Thatcher for "a disastrous failure to understand the seriousness and urgency of the situation . . . "

McDonnell, an active member of the Provisional IRA from Belfast, was serving a 14-year sentence for illegal arms possession after being arrested following a firebomb attack on a West Belfast furniture store in 1976. He was arrested and convicted with Bobby Sands, the first hunger striker to die in May, and replaced him in the protest.

Sinn Fein, the political wing of the IRA, announced that Patrick McGeown, 25, will replace McDonnell in the hunger strike campaign.