A 30-foot pleasure craft carrying Cuban refugees to Florida capsized today, drowning at least 10 of the 52 people aboard, as the Cuban government continued to force refugees onto overloaded American boats in Mariel harbor.

The U.S. Coast Guard rescued 38 passengers clinging to the sinking vessel. Half of them had no life jackets. Four persons are missing.

The accident -- the worst yet in the month-long "Freedom Flotilla" -- brought into sharp focus the dilemma of the Carter administration. The president could order the transfer of refugees onto safer U.S. Navy troopships outside Cuban waters, but doing so would sanction an exodus the U.S. government deems illegal.

Thus, the flotilla staggers on, as more than 4,000 Cubans a day, packed shoulder-to-shoulder on leaky, dangerous vessels, brave eight-foot seas in the Florida Straits. More than 54,500 refugees have arrived in the last 27 days aboard 957 boats.

The Coast Guard today strung a barrier of 10 ships linked by radar in a 200-mile area from Miami to Key West and succeeded in cutting off virtually all Cuba-bound traffic.

However, about 1,000 boats remain in Mariel, and their owners apparently are determined to rescue their Cuban relatives. Captains who might like to obey President Carter's order to return empty are being forced by Cuban soldiers to load refugees anyway, according to returning boat owners.

Today's accident, involving a Florida-registered boat called the Olo-Yumi, brought the flotilla's death toll to 20. Seven persons died in a severe storm at sea two weeks ago. Three died of asphyxiation on an overloaded boat Thursday.

Another vessel, the El Gallo, with 300 aboard, had been reported sinking early this morning off the Cuban coast, but returned safely to Mariel. Coast Guard officials said the report was a false alarm.

Despite the Coast Guard barrier and the seizure of more than 180 refugee-laden boats since Carter's get-tough policy was announced Wednesday, a small band of Cuban-Americans lingered along the docks today, determined to make a break for Mariel.

"I'm going anyway," said Salustiano Diaz, a house painter. "Even if they take my boat away and give me 10 years in jail, I'll go."

Diaz towed a gleaming 23-foot speedboat on a trailer behind his pickup truck all the way from his home in Metairie, La. He wants to rescue his 100-year-old mother and a brother who has been a political prisoner.

"I have not seen them for 15 years," he said. "All I ask is the opportunity to go and get them. They will not be a burden on the government. I will take care of them myself."

There was talk on the dock of organizing a 20-boat fleet to charge through the Coast Guard line tonight and then disperse rapidly in different directions.

"They won't be able to stop us all," said one Cuban-American whose 37-foot boat was loaded with six 55-gallon drums of gasoline. "They might be able to catch three or four of us, but the rest will get through."

The man, identified only as Alberto, had painted over his boat's name. He plans to leave at night, running without lights, and if the Coast Guard tries to board his vessel, he said, "I'll say I'm going fishing."

Coast Guard Cmdr. Samuel Dennis scoffed at the notion of a minirebellion. "Should they attempt to depart, I can assure you they will be stopped," he said.

Although the Coast Guard has been notoriously unsuccessful in halting the stream of small vessels that smuggle drugs into south Florida, Dennis said that with the reinforcements of the last few days, "I have five times as many resources now as I've had to stop drug smuggling in the past."

Confusion reigned today among the captains of the boats supposedly "seized" by U.S. customs. Vessels registered in south Florida are being allowed to return to their home ports, but may not be used after that, according to large red warning stickers plastered on their cabins.

Vessels from other parts of Florida and out of state are being directed to remain in local marinas. However, customs officials refused to say whether they will levy fines against boat owners or prosecute them once they return home.

The captains seemed for the most part unconcerned. "What can they do -- spank me?" said Genaro Chiroles, a Miami businessman whose shrimp boat brought seven of his inlaws from Cuba. "I'm sailing back to Miami and I'm going to take this sticker off."

Some boat owners speculated that the government would punish only those owners who had charged thousands of dollars in fees to Cuban-Americans desperate to retrieve relatives.

Meanwhile, many Cuban-Americans seem to have been deterred from making the trip by exaggerated reports that President Castro is sending criminals on the boats. Less than 2 percent of the new refugees are suspected of committing felonies, and boats in Mariel reportedly are still able to pick up relatives.