The U.S. Court of Appeals ruled yesterday that a federal law that prohibits discrimination against the handicapped in federally funded programs does not support sweeping government regulations designed to make public transportation fully accessible to disabled persons.

Instead of throwing out the regulations, however, the appeals court told the Department of Transportation to determine whether it could find support for them in other federal statutes that prohibit such discrimination in transportation services.

Otherwise, DOT might only be able to require local governments to propose plans to meet the needs of the handicapped.

The controversial regulations, approved by the Department of Transportation, are intended to keep handicapped persons in the "mainstream" of transportation systems. The regulations are now under review by the Reagan administration.

The regulations were challenged by the american Public Transit Association, which represents most of the nation's transit systems, and 11 individual transit systems, according to attorney Barry J. Cutler, who represents the association.

The appeals court said yesterday that while federal law bans discrimination, it does not mean that localities are required to take extraordinary steps, which would cost of billions of dollars, to accomodate the handicapped.

Relying on a 1979 decision of the U.S. Supreme Court, the appeals court said that a program might be considered discriminatory if a transit system refused to take "modest" steps to make transportation available to the disabled. The DOT regulations, however, require extensive modifications to existing transit systems and "impose heavy financial burdens on local transit authorities," Court of Appeals Judge Abner J. Mikva wrote in an opinion for the court.

Costs estimates for the progrm range from $3 billion to $7 billion. The federal government currently underwrites 80 percent of the program with subsidies, but, Mikva noted, there is no guarantee that such funds will continue to be available.

Mikva wrote that requiring local transit systems to make such "burdensome modifications" at great expense is beyond what was intended by a section of the Rehabilitation Act of 1973, which prohibits discrimination against the handicapped. Mikva was joined in his opinion by Judges Harry T. Edwards and George MacKinnon.

The appeals court said that the Urban Mass Transit Act of 1964 or the Federal Aid Highway Act of 1973 might give support to the DOT regulations, since both laws require that mass transportation systems must provide services that take into account the special needs of the handicapped.