A special federal court yesterday told creditors of the bankrupt Penn Central Transportation Co. and other former railroad enterprises in the Northeast that they are seeking too much money for property taken over in a government railroad reorganization.

In a long awaited decision on the value of rail properties conveyed primarily to the new Consolidated Rail Corp., a special court here indicated its belief that claims by the creditors are exaggerated but that the federal government's assessment of value is not adequate.

Using various formulas, creditors and trustees of the bankrupt railroad firms have estimated the values of assumed properties to be as much as $13 billion - all of which would have to paid from the federal Treasury if upheld in court.

But the U.S. Railway Association, a government planning agency, has pin-pointed the "net liquidation value" of the railroad properties at $621 million.

Federal legislation, which established Conrail to take over the Penn Central and other lines on April 1, 1976, required a transfer of rail properties at that time with valuation to be established at a later date. Otherwise, government planners thought the absence of agreement on price would prevent starting the new railroad.

If the government agencies involved and various creditor groups are unable to agree on valuation, litigation is expected to last for perhaps a decade or so and end up in a Supreme Court confrontation.

The special court yesterday zeroed in on this potential development and called on both sides to sit down and start reach an out of court agreement.

The Washington court was established under the Regional Rail Reorganization Act. The opinion yesterday established guidelines for reaching exchange prices on the properties which various bankrupt railroads turned over in 1976.

The opinion yesterday was written by the presiding judge, Henry J. Friendly, a senior judge of the U.S. Court of Appeals for the Second Circuit. Joining him in the decision were the other two judges of the special court: John Minor Wisdom, a senior judge of the U.S. Court of Appeals for the Fifth Circuit, and Roszel C. Thomsen, a senior federal judge for Maryland.

In testimony before the court, government lawyers concede that the rail reorganization involved a "taking" of any excess of the value of properties transferred to Conrail over the value of securities in Conrail issued to the railroads.

The principal questions discussed in yesterday's opinion were what kinds of evidence the bankrupt properties would be permitted to offer to show the existence and the amount of any such excess payment due.

Most of the creditors of Penn Central and the other bankrupt firms are other institutions.

The court held yesterday that both large bank, insurance companies or the government and the railroad's were entitled to offer evidence of what could have been realized if the properties in the properties had been sold as scrap. In addition, the court said railroads were entitled to show that larger amounts could have been obtained by selling all or part of the properties involved to solvent railroads or public agencies.

In emphasizing its decision about exaggerated claims, the court said it would not even consider "such fantasies of experts" as theories of social value or extensive depreciation "that would provide a bonanza to investors beyond anything of which they could have justly dreamed in the dark days of 1973."

The nation's needs for rail service afford no basis for awarding a value such properties did not possess for years and "which their owners could never have realized except through condemnation by the federal government," the court said.

The court told the parties to consider if their interests might not best be served by reaching agreements on a figure rather than asking the court and ultimately the Supreme Court for a resolution. The court said such a court decision would "necessarily unsatisfactory and perhaps to all among the conflicting opinions of experts as to what could have been obtained by way of sales with the attendant lapse of years and the incurring of enormous expense."

Cary W. Dickieson, chief lawyer for the U.S. Railway Association, said yesterday that his agency "in large part is rather pleased with the decision.