The Supreme Court created significant difficulty yesterday for the Reagan administration's plans to remove the federal government from the commuter railroad business when it ruled that states must follow federal law in negotiating with railroad labor unions.

The court held unanimously that New York City's Metropolitan Transit Authority, which owns the Long Island Rail Road, must follow the federal Railway Labor Act--and thus permit strikes--in contract discussions with the United Transportation Union.

The LIRR, which operates solely in New York and derives all but 5 percent of its revenues from commuters, had sought to apply a state law that, among other things, prohibits strikes.

The issue is particularly timely because New York, New Jersey, Connecticut, Pennsylvania and Maryland are deciding whether to assume control of commuter lines operated by Conrail, the federally organized northeastern railroad. New York's MTA already owns the LIRR, but three major lines from Westchester County into New York City are owned by Conrail.

Under a plan devised by Transportation Secretary Drew Lewis and adopted by Congress last year in budget reconciliation legislation, Conrail must leave the commuter business by Jan. 1 to make itself more attractive to a potential private buyer and to remove a drain on Conrail's management talent. Conrail, primarily a freight railroad stretching from New England to the Mississippi River, also says it has never been fully compensated by states and cities for running trains.

States must assume control of commuter operations, pay a new organization called Amtrak Commuter to run the trains or abandon the service, which carries hundreds of thousands of riders each day to Philadelphia, Newark, New York City and Washington.

Control over labor negotiations then will become a major consideration for the states, which already are less than enthusiastic about the Conrail situation. State authorities feel caught between a fear that Amtrak Commuter will be unable to handle Conrail's commuter operation, on the one hand, and a fear that their bus and subway employees will become disgruntled if they themselves take over the higher paid Conrail employees.

The court's decision appeared to rest heavily on the fact that the LIRR is first and foremost a railroad, and that federal regulation of railroads has a lengthy historical precedent, regardless of whether they are state-owned.

"It is certainly true," Chief Justice Warren E. Burger wrote for the court, "that some passenger railroads have come under state control in recent years . . . but that does not alter the historical reality that the operation of railroads is not among the functions traditionally performed by state and local governments."

The American Public Transit Association is concerned that the ruling will set a precedent for applying more generous federal labor standards to bus and subway operations around the country.

The Supreme Court is considering a Texas federal court ruling that San Antonio's transit authority did not have to follow the federal Fair Labor Standards Act in negotiating with its transit union.

"Certainly, one taking San Antonio's position wouldn't sleep more easily because of the LIRR decision but, on the other hand, I don't think it spells disaster," said APTA general counsel Robert Batchelder.

The decision disappointed those who had hoped the court would expand its 1976 ruling in National League of Cities v. Usery, which said Congress could not require state and local governments to pay their employees according to federal wage and hour standards. That ruling was hailed as an affirmation of the 10th Amendment, which reserves to the states those powers not specifically given to the federal government.

A.E. Dick Howard, professor of law at the University of Virginia, said, "Today's decision is one more case in a line of cases . . . limiting the reach of" the National League of Cities decision.

An NLC spokesman said, "We are extremely distressed. The thing we want to look at is whether the decision is being drawn strictly on national railroad policy or whether it signals a full assault on the Usery decision."