Boats jammed with asylum-seeking Cubans continued to sail into Key West, Fla., yesterday, as the State Department, seeking to halt a potential inpouring to thousands of refugees, threatened to prosecute boat owners who bring in illegal aliens.

Even as the department was announcing its threat, hundreds of boats, launched at five-minute intervals, were streaming from the Florida port en route to Cuba.

Department spokesman Tom Reston, in calling for immediate suspension of the impromptu sealift instigated by Florida's large Cuban community, warned that participants were "playing into the hands" of Cuban President Fidel Castro's communist regime.

But, U.S. officials admitted privately, it was the State Department, through its threats of prosecution, that appeared to be in the most immediate danger of falling victim to Castro's manipulations.

Specifically, the officials said, the situation was enabling Castro to divert attention from his disavowal of a U.S. agreement for the organized departure of some 10,000 Cubans in refuge at the Peruvian Embassy in Havana. He made it appear that the United States is unwilling to accept them.

The Cuban government managed that ploy by publishing a front-page article yesterday in the Communist Party newspaper Granma saying, in effect, that those in the Peruvian Embassy and, by implication, anyone else wishing to leave Cuba were free to go to the port of Mariel and board the boats waiting to take them to Key West.

The effect was to divert thousands toward Mariel in hopes of crowding aboard one of the boats rather than pin their hopes for departure on the risky assumption that Washington and Havana will be able to agree on orderly procedures in conformity with U.S. law.

Reston, noting that any Cubans who enter the country in this way are illegal aliens under the law, said:

"While we are deeply sympathetic with those in this country who want to expedite the departure from Cuba of those who are seeking freedom from Castro's regime, we cannot condone this procedure."

He warned the private skippers taking part in the sealift, and the Cuban exiles financing them, that they could face fines of up to $2,000 and jail terms of up to five years for breaking laws prohibiting "the transportation of undocumented persons to this country."

But reports from Florida said the warning was having no immediate effect on well-to-do Cubans from Miami, who continued to roam Key West's waterfront with fistfuls of cash, offering to buy or charter any vessel capable of making the 12-hour, 180-mile round trip and bringing their countrymen to Florida.

Six boats -- ranging from 25 to 70 feet and their decks crammed with Cubans -- completed the circuit yesterday, bringing approximately 450 refugees to Key West.

The first refugees to arrive were reported to be tired, hungry and, in many case, seasick. But all were clearly jubilant at being out of Cuba, and the Miami friends and relatives who had paid steep fees for their passage -- up to $5,000 per person, according to Reston -- openly told reporters in Key West that they considered it well worth the price and the legal risk.

Reston said the administration "deplores those who wish to profit from the plight of the refugees." The view in Key West was very different. One woman boat owner, who declined to give her name, told reporters she was charging $1,000 per passenger and added: "I'm risking a $60,000 boat to go over there."

Whether the State Department's warning of prosecutions would change the situation was not immediately clear last night. Reston refused to say what instructions had been given to the Coast Guard, and it was continuing to permit boats to dock and land passengers.

Reston did say that those Cubans who have arrived were being processed to see if they are eligible for entry to the United States. He added that those accepted will be counted as part of the 3,500 persons in the Peruvian Embassy that the United States had agreed to take in.

However, the spokesman left unclear whether this procedure would remain in force if the Cubans continued to pour into Key West in large numbers. He also did not specify what would be done with those who, for one reason or another, are adjudged ineligible for entry.

The situation began to unfold April 4, when Cuban guards around the Peruvian Embassy were removed, and hordes of disaffected Cubans began pouring into the embassy grounds, seeking asylum and demanding to leave the island.

After considerable diplomatic maneuvering, the United States and a number of other Latin countries agreed to take the refugees. Plans then were made to airlift them systematically to Costa Rica, where they were to be processed and sent on to the countries of their permanent asylum.

Last Friday, however, Castro, apparently stung by the humiliating publicity in the situation was attracting to his regime, suddenly cut off the Costa Rican air corridor after about only 800 of the regugees had left.

Since then, U.S. officials said yesterday, the United States, Peru and Costa Rica have been pressing Cuba to live up to the original agreement. But, they added, Castro Apparently saw an opportunity in the sealift instigated by impatient members of Florida's large Cuban community, and struck back with the unorthodox ploy now unfolding at Mariel.

By saying anyone can leave via the sealift, the officials conceded, Castro put the United States in the embarrassing position of having to invoke its immigration laws. That, they added, may obscure the fact the the United States offered to take 3,500 of those at the Peruvian Embassy and, in a pending 1980 refugee bill, is seeking permission for an additional 16,000 Cubans to enter the country.

Now, the United States faces a repeat of the situation occurred in 1965 when the Cubans permitted a stream of vessels from Miami to come into the port of Camarioca and haul away thousands of refugees under circumstances similar to those in Mariel.