Almost as soon as President Carter ad-libed his May 5 declaration that the United States would provide "an open heart and open arms" to refugees fleeing Cuba, the president's advisers began having concerns.

Carter's statement, a response to a question before the League of Women Voters, was not intended as a formal declaration of U.S. policy because, at the time, the administration had no policy -- only contradictory goals and hopes.

On the one hand, it wanted to stem the illegal boatlift of refugees, which threatened to overwhelm south Florida and lead to long-term economic problems where the new arrivals were eventually resettled.

On the other hand, the president was engaged in some diplomatic fencing with Cuban President Fidel Castro, a contest in which Carter sought to portray the United States in its traditional role as a haven for the oppressed, with Castro as chief oppressor.

Domestically, moreover, the administration was dealing with an explosive situation among Cuban-Americans intent on bringing their relatives to the United States regardless of the wishes or laws of the U.S. government.

As a result, for three weeks the administration frantically tried to deal with the problems piling up daily at Key West, Fla., while it stalled for time, hoping that the flow of Cuban exiles would slow or that Castro would agree to a more orderly procedure.

Instead, the situation grew worse, as more and more refugees came to South Florida and Castro began emptying jails and hospitals, sending the undesirables of Cuban society to the United States.

In the face of this, Carter yesterday made a formal declaration of U.S. policy. On the surface it seemed an abrupt reversal largely because of his apparently heartfelt but ill-prepared "open heart and open arms" statement of a week ago.

The president's announcement yesterday was put in terms of concern for the refugees' safety, and was accompained by an offer of a U.S. airlift or sealift of those Cubans who qualify for entry to the United States under the policy. But what Carter appeared to be trying to do was to shut the door on the mass influx of Cubans and to hand Castro the only key.

White House officials gave no indication that they expect Castro to use the key by accepting the U.S. conditions.

But they made it clear that Carter was hoping to place the burden and the blame on Castro.

"If Castro doesn't accept," said a senior White House official, "it will be apparent that it was all a ruse on his part -- not to reunite families, but to get people out of Cuba that he just did not want there anymore."

From the beginning of the three-week-old boatlift, the administration has been searching for a way to stem the massive and dangerous stream of exiles across the Florida Straits without setting off a possibly violent reaction in the large Cuban-American community in south Florida.

But their task was complicated by the president's statement, which suggested that the United States was ready to accept an unlimited number of Cubans.

Before his appearance at the League of Women Voters convention, Carter was given background material by his staff suggesting that he should speak of a humanitarian policy toward the refugees, but should stress that they would have to be dealt with in accordance with U.S. law, according to a senior White House official.

"We as a nation have always had our arms open to receiving refugees in accordance with American law," he told the convention.

But then, continuing his reply, he talked at length about America's tradition of accepting refugees, ending up with a promise: "We'll continue to provide an open heart and open arms to refugees seeking freedom from communist domination and from economic deprivation brought about primarily by Fidel Castro and his government."

Though his answer won applause it was no policy, and it complicated the task of those charged with fashioning one.

A few days later, Carter's advisers took their concerns to him. Domestic policy chief Stuart Eizenstat told the president that the boatlift was unsafe, that the refugees were straining U.S. domestic resources and that Castro, far from reuniting families, was exploiting the situation to rid his island of various problem elements, including criminials and low-income people.

Carter agreed, setting in motion a process that culminated in yesterday's announcement.

A key element that has changed over the last three weeks, giving rise to the hope that the crackdown ordered by the president will work, is the attitude of the Cuban-American community. At first, the fierce determination of these people to bring their relatives to the United States forced law enforcement officials to ignore the law and allow the illegal boatlift to continue.

But the Cuban-Americans, according to White House aide Jack Watson, have grown disenchanted with the boatlift because, for the most part, the refugees reaching Key West are not their relatives but people Castro wished to expel.

Calling the cooperation of the Cuban-American community "essential" to the success of the new policy, Watson said, "We could not have done a week ago what we've done today. Our judgment is that now is the time."

The administration, while holding out no real prospect that Castro will immediately agree to the U.S. proposal, hopes that a halt to the illegal boatlift will result in growing internal pressure on Castro to agree to the proposed U.S. airlift or sealift.

"If the boats stop, he'll have every incentive to cooperate," said one State Department official. "He'll have people backing up with no place to go. He'll be sitting on a powder keg."