FOR THREE years the United States government has been trying to sort out the difficult problem of the dangerous convicted felons Fidel Castro unloaded on the United States in the Mariel boatlift. That task has now been made substantially more difficult by an Atlanta federal judge who has demanded all manner of rights for these illegal immigrants and threatened to release them into the community if his demands are not met.

Back in 1980, it seemed to be a humanitarian and patriotic gesture to accept provisionally, without papers or visas, all those fleeing from the port of Mariel. More than 125,000 came. Most were true refugees, many had families here, and the great majority has settled into American communities without mishap. But the Cuban dictator played a cruel joke. He opened his jails and mental hospitals and put their inmates on the boats too.

This country has since provided care for the mentally ill among them. The 22,000 arrivals who freely admitted that they were convicts were kept apart and have been treated separately. Each case has been reviewed individually by Justice Department lawyers, and hearings were afforded at which the Cubans could be represented by lawyers and present witnesses. Gradually, those thought to be less dangerous, including not only political prisoners but also some persons who had committed serious felonies, were released.

Of the original group, however, 1,050 remain at the federal penitentiary in Atlanta. Some 144 of these have been approved for release and are awaiting sponsors. Those still in custody are, according to corrections officials, a difficult and violent population. Some have murdered other prisoners, and guards have been assaulted frequently. Each case continues to be reviewed on a regular basis. It would be simple enough to release them, but the Justice Department believes that this hard core--less than 1 percent of the Mariel boat people--is too dangerous to be allowed to roam the streets.

Now comes Judge Marvin H. Schoob. He concedes that deportation proceedings are civil, not criminal, and that the government has every right to hold illegal aliens pending deportation. But he has ordered the government to provide the Cuban inmates with free attorneys--though Americans involved in civil proceedings and other immigrants fighting deportation have no similar right. He has decreed that the Cubans should have subpoena power to summon witnesses--though no statute provides for granting such power. He threatens to release them unless these advantages are provided.

The matter could be settled quickly if these men could be deported, but Cuba refuses to take them back; the United States has tried to initiate negotiations on this subject, in vain, four times. As difficult as that may be, the pressure must be kept on the Castro regime to take them back. They belong in Cuban jails, not in American jails. Certainly they do not belong on American streets. Meanwhile, the government is under no obligation to accord these particular illegal aliens rights and privileges not available to other aliens or even to American citizens.