The Carter administration warned Navy leaders yesterday that they must clear up their present shipbuilding mess before they can expect to get enough money to build the big new fleet they want.

In the hardest-hitting speech yet made by an administration executive, Edward R. Jayne told a symposium at the Naval War College here that "present shipbuilding difficulties" are why President Carter did not give the Navy enough money to build its coveted 600-ship fleet.

Said Jayne, an associate director in the Office of Management and Budget:

"The multibillion-dollar cost growth, the delays of up to two years in delivery dates, and difficult relations between the Navy and its most important shipbuilders simply cannot continue in its present state."

Carter last week disclosed a five year program that would build only 70 new ships, fewer than half the number the Navy sought.

Jayne, who is associate director for national security and international affairs in OMB, warned that Navy leaders either must come up with a realistic, reasonably priced shipbuilding program or lose control of those decisions to White House and other non-Navy officials.

"Some in the Navy," Jayne told an audience here that included many Navy leaders, "seem more comfortable with continuing the rhetoric than with bailing out the bilge." The attitude of "damn the torpedoes, full-speed ahead" in asking for shipbuilding money "is not an acceptable response."

"If at this time next year we have not made visible progress in managing our shipbuilding program," Jayne said, the Army and the Air Force "will again receive priority" in the Carter defense budget because they "hold the promise of a greater return" on the money spent.

Carter's shipbuilding program will cost between $31 billion and $32 billion over the five-year period of fiscal 1979-1983, Jayne estimated. Defense Secretary Harold Brown said when the plan was revealed that it would provide the Navy with a fleet of 525 ships by the end of fiscal 1984, including older ones modernized to make them last longer.

A Navy study of what kind of fleet the country should have in the future calls for a more ambitious shipbuilding program than Carter has recommended to Congress. This study, released yesterday, will spark congressional debate over what kind of Navy should be built for the future.

Jayne took a swipe at that Navy study by asserting that "like many Navy studies before it," it argues for adding aircraft carriers and their escorts to fulfill the nation's requirement for projecting its power, controlling the sea lanes and making a show of force.

The OMB executive said that "I would be far more comfortable" if the Navy had spent as much effort examining such alternatives to carriers as land-based aircraft.

The Navy study, conducted under the leadership of Prof. Francis J. West Jr. of the Naval War College staff, states that, "Even with favorable technological trends, the overall fleet size is threatening to decline below the threshold of critical mass necessary for the containment of serious crises and the retention of fliexible options for the deterrence of major war."

The study says that a 3 percent annual growth in the Pentagon budget would finance a fleet of 579 ships. This is a larger fleet than the Carter five-year plan envisions.