Britain refused to join the American-led invasion of Grenada because Prime Minister Margaret Thatcher's government believed it to be unwarranted, dangerous in human and political terms and a gross violation of the sovereignty of a former colony.

According to senior Thatcher aides and Foreign Office sources, that judgment was reached at a lengthy meeting of senior Cabinet ministers Monday after assessing reports from Grenada and other Caribbean countries as well as weighing the potential consequences of British intervention.

The Cabinet group--including Defense Secretary Michael Heseltine and Foreign Secretary Sir Geoffrey Howe--concluded from intelligence reports that the situation in Grenada was "tense but calm" and seemed to be stabilizing. The risks to 200 British citizens there did not appear to be great. Both of those conditions would surely change for the worse if an invasion took place, the ministers agreed.

On the political front, they were concerned about the impact of such a move at a particularly sensitive moment in relations among the western allies, with the controversial deployment of medium-range nuclear missiles about to begin in Britain and elsewhere. They were especially mindful that morning because of the massive antimissile demonstrations in London and elsewhere over the weekend.

On this last point, one source said, "our worst fears were very quickly confirmed" by the denunciations of the United States throughout Europe this week and the renewed call in Britain for greater control of American nuclear weapons based here.

But most important, the government decided not to participate in the operation because its disapproval of the regime was not considered sufficient grounds for stretching international law and using military force to oust it.

"A change of government is not in itself sufficient reason to justify invasion by one country of another," Howe asserted in Parliament Tuesday, responding to charges that Britain had showed a failure of fortitude in not joining the U.S. operation and, alternatively, a lack of influence with the Reagan administration, in not being able to prevent it.

The prime minister's supporters contend that surprise here and in Washington over the British choice of restraint is based on the mistaken assumption that because of the Falklands conflict, "Thatcher spends her time waiting to go to war and that she would find it impossible to disagree with President Reagan on a matter like this.

"That view is seriously wrong on both counts," one source insisted.

Britain's role in the Grenada affair has also been confused, these sources say, by the notion that it must retain some responsibility for its troubled former colony, especially since the island's head of state, like Britain's, is Queen Elizabeth II and the monarch appoints a governor general there. The governor general, however, is chosen by Grenadans and Britain plays no part in any aspect of the island's administration.

Beyond their membership in the Commonwealth and the legacy of language and history, Britain has no special link to Grenada. British sources say that for reasons of geography and the presence of 1,000 of its citizens, the United States is far closer to the island in its sensitivity to the menace posed by the deposed government of Maurice Bishop and the now ousted revolutionary council that executed him.

For that reason, while declining to assist in the invasion, the Thatcher government decided not to condemn the United States for its intervention as other American allies in Europe did. Instead, Howe told Parliament that the American action was a "matter of regret" and attributed the U.S. move to differing "perceptions" of the proper course of action.

"We do not agree with the Americans on every issue," Howe said to choruses of derison from opposition critics, "any more than they always agree with us--nor are we expected to do so."

By neither supporting the American invasion nor assailing it, the government left itself wide open for a criticism that runs directly counter to Thatcher's hard-earned reputation for, as her campaign last spring called it, "the resolute approach." Her aides argue that it was not the position that was weak--it was straightforward and grounded in firm principles of respect for sovereignty--but the lackluster way it was presented.

This was the result of several separate misfortunes for Thatcher. The Foreign Office received its first word Friday night that the Organization of Eastern Caribbean States was considering a military intervention in Grenada and preparing to ask for help in carrying it out. But the seriousness of that notification was apparently underestimated because it was not rendered "formally."

The dispatching on Sunday of the destroyer HMS Antrim to the coastal waters off Grenada for the possible evacuation of British citizens was never meant to be a prelude to the sort of military operation that took place.

As events in the region unfolded over the weekend, British officials regarded the decision by CARICOM, the organization of Commonwealth countries in the Caribbean, to impose economic and political sanctions against Grenada, as the prevailing trend rather than the military option. They also accepted the Reagan administration's promises that it would consult London before taking any conclusive steps.

Both of these beliefs were incorrect.

In a press conference on Tuesday Secretary of State George P. Shultz revealed that President Reagan made a tentative decision to invade on Sunday evening, but no word to that effect was sent to the British for another 24 hours--only a short time before the final go-ahead was given.

So by the time that the Cabinet met Monday morning, the British case was essentially irrelevant as the U.S. military gears were in quickening motion. It was this failure to "stay in touch" with London, as had been promised, that was particularly annoying and embarrassing to Thatcher.

Moreover, Howe looked foolish when his assertion to Parliament Monday that the United States had no intention of invading Grenada was dramatically contradicted a few hours later. This highlighted the government's ineffectual appearance.

"Howe's dilemma was to demonstrate our differences with the Reagan administration without lambasting it, and that made matters worse," one source observed.

A senior government official said today that Howe made all these points of grievance forcefully in a meeting with Shultz in Paris yesterday and received the secretary's apologies.

But British diplomats in the region must also share the blame, sources say, for failing to perceive that the CARICOM meeting consensus on sanctions was a smokescreen by the eastern Caribbean states, plus Barbados and Jamaica, which were pressing for an invasion. The diplomats were not aggressive enough and asking all the right questions, a politician observed.

On balance, therefore, Britain's effort to take a responsible position on the rights of a tiny land thousands of miles away backfired badly. Relations with the United States were strained; Thatcher's standing as a formidable player in the international arena was diminished, and Grenada was invaded, precipitating many of the consequences she had warned against.