The Carter administration started to get tough today with boat owners bringing refugees from Cuba, and one of the first to feel the impact was Jesus Cruz.

As his shrimp boat, the Judy Ann T, put in to port today with a standing-room-only load of 300 refugees, Immigration and Customs officials on the dock informed him that he could be subject to $600,000 in fines for his humanitarian trip to Cuba.

The potential fine, and others like it, are the device by which Washington hopes to choke off the steady flow of private boats, many of them tiny, fragile craft that have been shuttling the 90 miles to bring back an ever-growing number of refugees.

Although the weather has been clear, the Coast Guard reported that at least five boats have sunk on the way out and many others have run aground or have been towed back to port. No one has been reported injured. The flotilla, according to aerial surveys, involved more than 1,000 vessels either en route or already in Cuba. As of late afternoon yesterday, only 17 had returned.

Thus, while refugee-landing joy reigned around him, Cruz stood under a piercing morning sun and heard the news that he could be fined $1,000 for each passenger brought here illegally and another $1,000 for each one who does not show up for more immigration processing Saturday morning in Miami -- about three hours north of here.

That meant, unlike previous cases, the federal government would no longer provide housing or transportation for the newcomers. It also left Jesus Cruz with a $600,000 Excedrin headache.

But as in other instances, the Cuban-American community responded quickly. Veterans of the Bay of Pigs invasion sent word that they would provide buses. The Latin Chamber of Commerce here provided food and drink. Overnight housing was being sought in Miami.

The federal ax was falling on one of the few commercial boat operators who has not made a bundle off of the hundreds of desperate Cubans who have come here to try to finance trips to bring relatives out of Cuba. Fees run $1,000 or more for each exile traveling to get relatives.

Cruz arranged this trip on his big blue-and-white Judy Ann T. for his own relatives and the family of his exile friend from Miami, Alfredo Garcia, charging nothing.

One of Cruz's friends, Ramon Gonzalez, a young Miami real estate salesman, worried about the penalties, but not too much. "I don't know how we'll deal with this," he said, "but I don't care if they fine me or not. Man, I brought back my three brothers and their families that I hadn't seen for 12 years."

But Cruz and the Garcias did not pick up all the relatives they had gone for, and today, even as the possibility of heavy fines faced them, they were considering a return trip to Mariel, the Cuban port about 90 miles south of here.

Since authorities started counting, 1,312 exiles have gone through the immigration procedures, many of them part of the 10,000 who stormed the Peruvian Embassy iin Havana to seek asylum earlier this month. They are being given 60-day deferred inspections that allow them to get food and aid and even to seek work before applying for formal asylum.

The State Department reported that letters and telephone calls were running heavily against the boatlift, many persons protecting that the Cubans were getting unfair preference over other aliens. The office of Sen. Lawton Chiles (D-Fla.) said constitutent response had been 80 percent negative.

That seemed to bolster administration determination to be as discouraging as possible to the boatlift. "We're getting cranked up now to really enforce these laws," said one State Department official.

Immigration officials were reported to be fanning out to the dozens of small airstrips around Miami in hopes of spotting any exiles coming in that way. One two persons had come in by air as of last everning, but a Miami radio station urged private pilots to get going. Havana authorities reportedly cleared a dirt strip for exile use.

Travelers returning yesterday, reported that 600 or more small craft were jamming the docks at Mariel, awaiting clearance and loading of those Premier Fidel Castro will allow to leave.

Under his procedures, the exile crews must take out four other Cubans for each relative they seek to pick up at Mariel. Local radio stations were saying today he may allow as many as 250,000 to leave.

State department officials said they know 200,000 want out. "Castro is doing what any pirate would do, basically making them walk the plank," said one. "It's like what Pol Pot did in Cambodia," he said.

State Department officials briefed congressional leaders and their staffs yesterday afternoon while the White House invited members of the Cuban-American community for an exchange of views Saturday. "It's not clear how long they'll be willing to finance the newcomers, one harassed official said.

The jobs and backgrounds of the first refugees are not yet known. Some sources said they feared Castro would make leaving more difficult for professionals holed up in the Peruvian Embassy than for less skilled workers. Many boat owners are apparently coming without the family members they went to get.

The uncertainty has spawned massive confusion, which in turn has made exiles desperate in their efforts to bring loved ones from Cuba. For their part, many refugees on the Judy Ann T. talked emotionally about leaving wives, children and parents behind.

Desperation, U.S.A.: Mrs. Perrot Bombino, a Cuban who arrived in Miami in 1971, and her American husband decided it was now or never to get her 11 relatives waiting in Havana. They bought a 24-foot boat for $3,700 towed it to Key West, put it in the water and then tried to figure out how to make it work. Neither had ever operated a power boat.

"We collected money from friends and family to do this," she said. "I hope we make it. But I'm so scared, I'm scared to death. My husband says it we can only bring one, we bring only one." Desperation, cuba: Felipe Flechet, 26, possessor of only the clothes he wore, stepped off the Judy Ann T. happy to be here, and said that his wife and three children weren't with him.

"I just couldn't risk bringing them it was too dangerous when we hid in the Peruvian Embassy. Now I will make every effort to bring them over here somehow," he said.

But like others arriving here, Flechet said that deprivation and political persecution have reached such levels in Cuba that the risks involved in leaving seemed mild.

"I am a graduate labor organizer. After two years, my salary was cut from $192 a month to $138, when they started making me pay back my tuition costs. Consider that these shoes I wear cost about $80. How can you live?" he said.

Fletchet, a black, said that Cuban blacks continue to suffer a "subtle discrimination" that dims career opportunities but, he added, his race may have been less of a handicap than his refusal to serve with Cuban troops in Angola.

"I spent three years in the army," he said, "but I absolutely refused to go to Angola. Fight in another country? No sir. After I was discharged in 1974, that haunted me -- all the doors of employment were closed."

Noting the large numbers of young men who have come here this week, Fletchet said that mothers are encouraging sons to leave Cuba and find a better life abroad.

Jesus Valdez, 20, a technician; Juan M. Semanat, 27, a hydraulic engineer; and Lazaro Bijande, 18, a physical education student and amateur boxer, stood listening on the dock, nodding heads in agreement.

"We have left loved ones," said Semanat, "but this trip has given us all something new. We are a family now."