Navy ships scheduled to take part in an amphibious landing at Guantanamo Bay next week are being diverted to help the Coast Guard monitor the flood of Cuban refugees in the Florida straits, the Pentagon announced last night.

President Carter ordered the shift "to save human lives and ensure maritime safety the unprecedented emergency," according to a statement by Adm. Harry D. Train II, commander of the Atlantic fleet.

The amphibious landing was canceled in part to make propaganda points for the United States in its battle of one-upmanship with Cuban President Fidel Castro, according to one official involved in the White House deliberations. "Someone suggested turning the Navy into a rescue operation instead of a show-of-force operation," the official said.

Meanwhile, in Key West, Fla., about 50 boats carrying 1,500 Cuban refugees arrived in the last two days, bringing the total number of refugees to more than 5,000 in the last two weeks. The wave of immigrants could reach 20,000 within a few days, authorities said, but there is still no organized program to house or resettle them.

About 1,700 boats are anchored in the Cuban port of Mariel, waiting to ferry more refugees across the 90-mile span between Cuba and the Florida Keys. With the weather clear again after severe storms early in the week, it was expected that the arrival rate would increase in coming days.

A Pentagon spokesman said last night that five amphibious landing craft and some armed escorts slated to take part in the Guantanamo Bay landing would be deployed instead to reinforce the Coast Guard's effort to aid small boats in the so-called "freedom flotilla."

Because of "this urgent humanitarian need," the Guantanamo amphibious exercise in Navy maneuvers called Solid Shield 80 is being canceled, the Pentagon announcement said.

Carter had called for the show of force near Cuba last fall in response to the discovery of Soviet troops there.

Cuban officials reportedly considered the planned landing by Marines an affront to national pride, and reports from Havana said Castro has been considering opening a second port -- near Guantanamo -- for Cubans seeking to emigrate in an apparent effort to complicate the scheduled mock assault.

Administration officials said last night that the possibility of a second refugee port wasn't a factor in the decision to cancel the landing. "I never heard of another port," one said.

At a hearing before the House Judiciary Committee yesterday, Victor H. Palmieri, the State Department's coordinator for refugee affairs, said the administration was considering several options in an attempt to bring the refugee flood under control, including some form of direct or indirect negotiations with Cuban government officials.

But the administration's unsettled policy was clear in several of Palmieri's statements.Among the items he discussed:

The administration intends to have a processing center set up within 24 hours to screen out criminals and other undesirables who have been joining the refugee tide, but he said he didn't know exactly where the center would be located.

The administration is considering holding a conference with Latin American allies in the near future to consider an international solution to the problem.

Special legislation may be offered to resolve the legal status of the arriving Cubans, who now are being processed as applicants for political asylum.

In his prepared statements, Palmieri said "like scores of first-asylum countries around the world today, we will be generous; we will be sesitive to the basic human desires that motivated their flight; no boats will be turned away, and no one will be returned to a country where he or she might face persecution."

Asking during a break in the hearing if this "open arms" statement didn't contradict previous administration pronouncements taking a hard line on the flow of Cubans, Palmieri said, "There are changing circumstances. We can no longer be the sole destination of such flight."

He said the administration intends to deport criminals who have infiltrated the flood of exiles and to find places abroad for people who don't have family ties in the United States. "Doesn't that make perfect sense?" he said with a smile. Then he added, "I'm trying to make less than perfect sense."

David Crosland, acting commissioner of the Immigration and Naturalization Service, said that he wanted refugees with criminal records screened carefully to make sure that some hadn't been jailed for political crimes, such as speaking against Castro's government.

Palmieri also acknowledged that the government's enforcement effort to stop the illegal boat armada has been selective. Noting that the emotional reaction of the Cuban community to the immigration controls might cause "major disobedience or civil disorder," he said, "Our enforcement should be aimed against cases of profiteering." He said three boats have been seized by federal authorities.

Despite the seizures, and concern that the government might impound more boats and impose heavy fines, Cuban-Americans in Florida continued their frantic effort to fetch relatives. About 200 boats left Florida ports for Cuba yesterday, many of them too small to make the voyage safely, Coast Guard officials said.

Coast Guard cutters and helicopters criss-crossed the Florida Straits yesterday, responding to 250 distress calls. They towed in several disabled vessels, but found no trace of the boaters from more than a dozen craft found capsized after a violent storm last Sunday. At least four men involved in the refugee boatlift died in that storm. Authories believe the toll may have been higher, but no other have been recovered.

Harried officials, many of whom are working nearly around the clock, convey a sense of impending disaster unless federal relief comes soon. "We have adequate controls to move a thousand people a day to Miami," said Ron Villella, who is directing the state operation here. "If 18,000 people show up on that dock in the next 48 hours, we'll be in serious trouble."

National Guard armories and other shelters have been set up in Miami to temporarily house the refugees, but no one has yet found permanent housing or jobs for them.

However, state officials are urging the INS not to take enforcement action against boat captains because, as Villella said, "if word spreads that the captains are being fined, they'll drop these people up and down the Florida coast."

Meanwhile, many boats were returning from Mariel empty after waiting days without food and drink. Cuban officials told the captains they could return to pick up relatives in a few weeks.

The scene on the dilapidated former Navy base at Key West, where the refugees are disembarking, is part chaos, part carnival. Politicans, priests, doctors, customs officials, immigration officials, journalists and volunteers pepper the bedraggled refugees with questions as they tumble off rusty shrimpers and mall outboard craft.

Rep. Elizabeth Holtzman (D-n.y.), a member of the House Judiciary Committee, toured the docks this morning. "Just about every adult male I have talked to admitted to having been in prison in Cuba," she said.

Holtzman, who is running for the U.S. Senate in New York, said, "We have to have extremely careful screening procedures. We have to know the extent to which Castro is opening up his jails."

Ray Morris, district director of the INS, said the United States has detained 17 refugees who admitted to having been in jail in Cuba.

"We don't know how many social deviates are coming in," he said. However he said that 90 percent of the refugees are "pretty much like the rest of society in America. Some hardworking, some lazy. They strike me as normal human beings."

According to INS, a quarter of the refugees have immediate relatives in the United States and 44 percent have distant relatives here. Their average age is 31. One out of five is under 20. Nine percent are black.