A federal judge told Metro yesterday that it cannot spend any money on construction of the disputed Green Line to Rosecroft in southern Prince George's County unless it goes through the entire planning process again.

The order, signed by U.S. District Court Judge Norman P. Ramsey in Baltimore, enjoined Prince George's County and the Washington Metropolitan Area Transit Authority (Metro) from any expenditures for a rapid transit rail line that fails to conform to a plan adopted by county voters in 1968. That original route calls for construction along Branch Avenue, several miles north of the Rosecroft alignment.

Metro spokeman Cody Pfanstiehl said last night he had not read the ruling, but he said, "Obviously it puts us back to square one, in 1968."

Pfanstiehl said the decision technically would not affect the construction of that section of the line between the Navy Yard and Greenbelt, but he added that the uncertainty of the alignment "puts us into an extremely difficult funding situation" because of Reagan administration cutbacks in funding for mass transit.

Yesterday's order was the latest development in a two-year-old lawsuit aimed at stopping the construction of the Rosecroft segment, known officially as the F route, which would link the Navy Yard in Southwest Washington with the Rosecroft Raceway in Oxon Hill. The route is a portion of the long delayed Green Line, the last segment of the 101-mile system scheduled for construction.

The lawsuit was filed after the County Council voted in 1978 to change the F route terminus from Branch Avenue to Rosecroft, saying the Rosecroft route would be cheaper to build, cheaper to operate and would provide better geographical service.

Two groups of business people joined forces to sue the county and Metro: a Branch Avenue car dealer who claimed he built a larger showroom than now necessary in anticipation of the increased business; the other, owners of two apartment houses near Rosecroft who feared the disruption of the neighborhood by the rail construction. They contended that the county failed to give voters adequate notice of the change, and further charged that the change was politically motivated.

Thomas Duley, one of the attorneys who brought the suit, said of yesterday's ruling: "I think the effect is that unless they build the Branch Avenue line, I don't think south county is ever going to get a subway. As a lifelong resident of the county, I'd like to see the line built but if they insist on trying to move it to Rosecroft it's dead. We'll keep it tied up in litigation for 20 years.