A group of foreign students at the University of Maryland, who have been trying since 1974 to be declared state residents and thus take advantage of lower in-state tuition rates, were set back in their efforts yesterday by a decision of the United States Supreme Court.

The high court yesterday vacated an appeals court ruling in favor of the nonimmigrant aliens and sent the case back to U.S. District Court in Baltimore for a rehearing.

The appeals court had ruled that the university's refusal to grant favored "in state" status to these students denied them the constitutional guarantees of due process.

Many of the students affected by the suit hold "G-4" visas as the children of officers or employes of such international associations as the World Bank, and have lived in Maryland for years.

An alien student entering the university as a freshman in 1978 would have paid about $1,600 more in tuition than a state resident, according to Assistant Attorney General David Feldman. The case affects about 60 student at the university, Feldman said.

The Supreme Court held yesterday that because the university's Board of Regents in 1978 clarified the status of these aliens the "posture of the case" had changed, and it should be reheard in District Court.

The students' attorney, Alfred Scanlan, said yesterday he still believes the university policy is illegal, but he does not know whether the students will want "to engage in five or six more years of legal warfare."

He said he will argue that the university's "clarification' of its policy which was made about two years after the U.S. District Court ruled in the students' favor, cannot be made retroactive. Scanlan said it should not apply to students who paid the higher tuition rate between 1976 and 1978.