Their countrymen who stayed behind called them gusanos (worms) because of their eagerness to escape the communism pervading their native Cuba. But for many of the 125,000 Cubans who came here in the 1980 Mariel boatlift, the flight to freedom led instead to a new life behind the chain-link fences of American detention centers. Now it threatens to come full circle with their forced return to their homeland.

That is the prospect facing at least 2,500 of those refugees officially classified by the State Department and Immigration and Naturalization Service as "Mariel excludables." They are people whose criminal records or histories of mental illness normally would have disqualified them for entry into the United States and who thus are subject to a U.S.-Cuban agreement calling for their repatriation to Cuba.

Moreover, these 2,500 may be only the first wave in a series of deportations that eventually could see more than 7,000 Marielitos sent back to Cuba.

The riots that broke out at INS detention camps in Oakdale, La., and Atlanta in the wake of last Friday's announcement of the agreement with Havana has underscored the panic-stricken desperation of those who face deportation.

Their plight began in the spring of 1980, when thousands of Cubans, driven by badly deteriorating economic conditions, swarmed over the grounds of the Peruvian Embassy in Havana, demanding political asylum and the right to leave the country. The incident, which attracted worldwide attention, prompted Cuban President Fidel Castro to announce that anyone who wanted to leave was free to go.

However, when other Latin American countries refused entry to the thousands of Cubans clamoring to get out, Castro announced that the United States would have to take them and ordered the would-be departees to exit through the small coastal port of Mariel.

There, over the next several weeks, a seemingly endless convoy of small boats bought or chartered by Cuban refugees in this country moved in and out, transporting what eventually totaled 125,000 men, women and children across the 90 miles of Atlantic waters to Key West and other Florida ports.

The Carter administration was unable to stem the invasion and found itself confronted with an army of refugees who had arrived on U.S. shores without visas or documentation.

Unable to send them back as it normally would do with illegal aliens apprehended at the border, the Carter administration gave the newcomers permission to remain in the country temporarily and seek to make their status permanent at a later date. In fact, the vast majority of those who came from Mariel in 1980 are eligible to apply for U.S. citizenship this year.

But, within this larger mass, one group could not be absorbed under U.S. law. At the height of the boatlift, Castro, thumbing his nose at the United States, emptied many of Cuba's prisons and mental hospitals and put the occupants, together with many others that his government regarded as undesirable, onto the vessels leaving Mariel.

U.S. officials thus were confronted with a large number of people who did not qualify for entrance into the country because of criminal records, mental illness, serious health problems or sexually deviant behavior and who therefore had to be categorized as illegal aliens.

In addition, a large number of other refugees later committed crimes in the United States, violating the temporary parole under which they had been admitted.

According to INS officials, only 210 had to be detained immediately upon arrival. But many of the others soon began putting heavy strains on the police and welfare resources of Miami and other cities, causing demands that they be sent back to Cuba.

In December 1984, the State Department negotiated a comprehensive emigration accord with Cuba that included Havana's agreement to take back 2,747 Marielitos who U.S. officials contend were found to be ineligible to stay here. But, in mid-1985, after 201 on that list had been deported, Castro suspended the agreement to protest the Reagan administration's opening of Radio Marti broadcasts to Cuba. It is the 1984 agreement that Washington and Havana agreed to reactivate last Friday.

In the meantime, the INS found it necessary to put large numbers of those it regarded as potential deportees into detention centers established at Oakdale, Atlanta and other locations, including St. Elizabeths Hospital here. A class-action legal suit was brought on behalf of some detainees, but in 1983 the 11th U.S. Circuit Court of Appeals ruled that as aliens they had no constitutional rights. The Supreme Court last year declined to review that decision.

The upshot is that the 2,500 people specifically covered by the U.S.-Cuban accord can be deported immediately. However, while the State Department reportedly believes that these are the only people to be deported, Justice Department sources said yesterday that they regard the agreement as open-ended and applicable to as many Marielitos as the courts determine to be excludable.

Duke Austin, an INS spokesman, said the possibility of the number exceeding 7,000 is based on the fact that the INS currently has 3,751 Mariel Cubans under detention. He added that another 3,830 are in state and county prisons around the country for crimes committed in the United States and will become INS detainees when they finish their prison terms.

Of those already under INS detention, 1,393 are in the Atlanta center, 998 in Oakdale, 117 at St. Elizabeths, 304 in other INS facilities and 939 held under contract in county and state jails.