Two recent federal court rulings have set the stage for far-reaching changes in one of the most controversial, potentially costly and often emotional areas of U.S. immigration policy--the constitutional rights of undocumented aliens arrested while trying to enter the country illegally.

The Reagan administration has contended that these "excludable aliens" should be detained indefinitely if the Justice Department determines that such action is warranted.

The recent court rulings acknowledged that the aliens are in this country illegally and may be awaiting return to their homeland or admission to the United States. And they acknowledge that the aliens' rights to due process when they seek admission are limited to those set forth by law, not the constitution.

Nevertheless, the courts said that while in this country aliens, whether legal or illegal, enjoy constitutional protections from racial discrimination and from indefinite imprisonment without a proper hearing.

In April, a federal appeals court in Atlanta upheld a lower court decree that ordered the government to release 1,700 Haitians who arrived here in 1980.

The court contended that illegal aliens in similar situations from other countries had not been detained indefinitely, and that the new policy was applied almost exclusively--and in discriminatory fashion--to Haitians.

Then on July 7, U.S. District Judge Marvin H. Shoob of Atlanta ruled that 1,050 Cubans being held in Miami could not be detained any longer unless the Justice Department provided them with lawyers and proved to neutral observers that, as the department contends, the aliens are too dangerous to be freed.

The department has asked the the 11th Circuit Court of Appeals in Atlanta to reconsider its April ruling on the Haitians but has not decided whether to appeal Shoob's decision.

Some lawyers familiar with immigration law see the two rulings as giving new rights to illegal aliens.

"What makes this a landmark decision is that you can't say that when it comes to aliens like these we can simply throw them in the hoosegow and throw away the key," said Maurice C. Roberts, former chairman of the U.S. Board of Immigration Appeals and now editor of Interpreter Releases, a Washington-based newsletter on immigration.

The rulings do not apply to the larger category of illegal aliens, an estimated 800,000 per year, who are apprehended and deported, often to reenter a few days later.

Under normal conditions, only about 1,000 illegal aliens are detained after being apprehended. But that category swelled to several thousand in 1980 and 1981, with an influx of 125,000 Cubans in the Mariel boatlift and an estimated 50,000 Haitians.

About 100 Haitians are now in detention, most apprehended recently. The 1,050 Cubans are the only ones remaining in detention from the boatlift days. The U.S. government wants to send them back to Cuba, but Cuban authorities refuse to take them.