President Carter moved yesterday to shut off the illegal benefit of Cuban refugees to the United States, but offered to provide government transportation for Cubans seeking reunification with their families in the United States, if Fidel Castro agrees.

Acting after the three-week flow of refugees neared 40,000, Carter announced that beginning immediately the United States will accept only Cubans with relatives in the United States, Cuban political prisoners and those Cubans who have taken refuge in the U.S. diplomatic mission and the Peruvian Embassy in Havana.

Last night, the State Department announced that 17 U.S. diplomats and dependents were being withdrawn temporarily from Havana because of a "virulent and continuing anti-American campaign" in Cuba.

An Eastern Airlines 727 bought the Americans to Miami early today.

A State Department spokesman said, "We simply think it would be irresponsible for us not to pare down our staff" at the U.S. Interests Section in Havana.

In his effort to stem the flow of Cuban refugees into this country, Carter ordered the Coast Guard and law enforcement agencies immediately to begin leveling fines and seizing the vessels bringing Cubans to the United States, including those boats that were at sea yesterday as part of the boatlift between Cuba and Key West.

Carter said the United States will provide air or sea transportation to Cubans who qualify for entry to the United States under this latest policy, but only if Castro first agrees to allow American officials to screen potential refugees in Havana to assure that they fit into one of the four categories.

Carter acknowledged that he has no assurance that the Cuban president will agree to these conditions, which seemed unlikely, and White House officials said that in that event the government will "enforce the law vigorously" in an effort to halt the Cuban influx.

"In the absence of Mr. Castro's agreement to what we are proposing, we are determined to stop the current disorderly boat flow," said president assistant Jack Watson.

Late yesterday, the Coast Guard began making hourly radio transmissions to all boats in the waters between south Florida and Cuba, warning American boat owners to return to the United States without refugees or face fines and seizure of their vessels.

Watson said the number of boats involved in the boatlift yesterday was not known, but he estimated that there were 1,400 boats at the Cuban port of Mariel, where the refugees board for the hazardous 90-mile trip to Key West.

Boats continued to pour into Key West, where they were quickly impounded. On one boat, three passengers were dead of apparent carbon monoxide poisoning. Two more refugees were taken to a hospital.

"Everyone below deck was piled on top of one another and the exhaust from the engines was running through there," said Lt. John Kirchner, captain of the Coast Guard cutter Point Huron.

Attorney General Benjamin B. Civiletti said boat owners who defy government warnings and persist in bringing refugees to the United States will be prosecuted.

Unless Castro, in a sudden reversal of attitude, agrees to cooperate with the proposed U.S.-government-sponsored airlift or sealift of refugees, the net effect of Carter's actions yesterday would be a heavy crackdown on the illegal boatlift, stemming the flow of refugees and risking the wrath of the Cuban-American community.

White House officials, however, predicted a "good chance" for support of the program by Cuban-Americans, who, they said, have grown disenchanted with the boatlift because of the risks involved and because Castro is emptying Cuba's prisons but doing little to reunite families in the United States.

"The goals of the Cuban-American community are not being met," said White House official Stuart Eizenstat. "Castro is exploiting the situation by sending out the people he wants to send out."

Carter's action yesterday will have no effect on the Cubans who already have reached the United States or those who enter the United States later, even on boats then seized for participating in the illegal boatlift. The status of each Cuban will be decided on an individual basis, and those deemed to have a genuine fear of persecution if returned to Cuba will be allowed to stay.

Officials said there will be no immediate "blanket determination" that the Cubans, or the refugees from Haiti who are also streaming into Florida by boat, qualify as political refugees and are therefore eligible to receive federal assistance for two years.

This question and others including potential costs to the federal government, as well as state and local governments where the new arrivals are resettled, was left open for future consultation with Congress.

Nor would the officials estimate how many Cubans would be eligible for entry into the United States under the new policy should Castro cooperate. There have been reports that as many as 250,000 Cubans are seeking entry into the United States, but Watson would say only that it could be "a very large number."

White House officials yesterday also announced additional steps to deal with the Cuban influx, including:

The opening today in Miami of a family registration office to begin collecting the names of Cubans with relatives in the United States who qualify for entry under the new policy.

Plans to open a third processing center for the Cubans at Indiantown Gap, Pa., with an eventual capacity of 20,000.

Watson said that, as of yesterday, about 39,000 Cubans had reached Key West by boat, and that 17,000 had been resettled. He said the processing center at Eglin Air Force Base, Fla., was at its capacity, with 10,000 refugees, and that 8,000 Cubans were at the second processing center, at Fort Chaffee, Ark.

Suggesting that the administration expects the flow of refugees to continue at least for a while even with the threatened fines and boat seizures. Watson said he expected that the Fort Indiantown Gap facility will have to be used when Fort Chaffee reaches its capacity of 20,000 refugees.

In a statement, the White House also accused Castro of taking "hardened criminals out of prison and mental patients out of hospitals, and has forced boat owners to take them to the U.S.

"We will not permit our country to be used as a dumping ground for criminals who represent a danger to our society, and we will begin exclusion proceedings against these people at once."

Watson said about 400 criminals have been detained in south Florida, and that their disposition will be determined in consultation with the U.N. High Commission on Refugees. He acknowledged that their return to Cuba also hinged on Castro's cooperation.

Watson said the purpose of the president's action is to provide "an orderly and safe flow" of the Cuban refugees, and to relieve a system of dealing with the refugees that is "totally disorderly." He said that there have been seven deaths in the boatlift, and warned of "great loss of life" if it is not stemmed.

State Department spokesman Mark Sakwoski said last night that the withdrawal of some U.S. personnel from the U.S. mission in Havana "does not mean a break with Cuba . . . We think it is prudent due to the circumstances of the anti-American which has been whipped up."

A nationwide anti-American demonstration has been called for Saturday. Part of the route is to pass the Swiss Embassy in Havana, where the U.S. Interests Section is located.

Spokesman David Hall said last night that State Department officials hoped the withdrawal of U.S. personnel would be a temporary measure.

Hall said the United States holds the Cuban government responsible for the protection of the U.S. mission, "everyone in it and all U.S. diplomats in Cuba."

Sakowski could not say how many U.S. diplomats, and personnel were remaining in Cuba.