THE REAGAN administration finds itself tied in an uncharacteristic knot in the matter of taking in some 3,000 former and current political prisoners in Cuba. Fidel Castro, whose policy is to throw out socialism's misfits, is ready to let them go. You might think that this administration would leap to offer relief to people who are presumably deeply anticommunist and have suffered for it. Yet the administration hesitates.

What has happened is that the package that included the prisoners disintegrated, and the administration is still trying to put it back together. The package was the agreement of 1984 in which the United States finally got Fidel Castro to agree to take back the unassimilable criminals and mental cases he had cynically slipped into the boat lift of 125,000 Cubans whom he had let sail from Mariel in 1980. Jimmy Carter, under the immense pressure generated by the ''freedom flotilla,'' had suspended the regular screening of would-be immigrants. Upon discovering the hard-core ''excludables,'' he then suspended all further Cuban immigration. By 1984, officials had worked out a deal for release of the prisoners, a resumed flow of regular immigration and return of the excludables. But the Castro regime took the occasion of the startup of Radio Marti, an official propaganda station beaming to Cuba, to denounce the immigration accord.

An initiative launched, and coordinated with the State Department, by Edward Kennedy helped put things back on track. The senator has been working off a family debt to free all the prisoners taken at the Bay of Pigs; the last commander came out a month ago, and only one soldier now remains. From this project, the immigration package was revived, with one new element. To enable Mr. Castro to swallow his defeat in failing to force the United States to take Radio Marti off the air, the Americans agreed to help him do what he could have done anyway -- broadcast his propaganda to the United States. In talks last week, however, he made such absurd demands -- they would have required the United States to break radio law and squelch dozens of American stations -- that the immigration package was derailed again.

Did Mr. Castro feel he could get rid of misfits -- his apparent priority -- without paying the price of taking back the excludables? Whatever, voices on the American right are demanding that the Reagan administration break the political prisoners out of the package and take these worthy people in. The administration fears this would leave it with too little leverage to pry open the regular emigration door and to send back the excludables. Few dilemmas of policy carry such a heavy emotional weight. It is a judgment call for a Solomon.