Pentagon leaders have mounted the biggest challenge yet to the iron rule of Adm. H. G. Rickover, the 79-year-old czar of the Navy's submarine program.

The design, performance and cost of the Rickover submarine fleet are suddenly under challenge to an unprecedented degree.

Impetus comes from stunning Soviet Rickover preserve; soaring costs and construction delays that confront the United States with gaps in its missile and attack submarine forces, and an eroding political base for Rickover.

"Rickover has been implacable" when it comes to promising technical innovations, said one Pentagon executive who wants to replace the nuclear power plant Rickover advocated for the Trident submarine specialist. He complained that the Navy has no program for building submarines, as the Soviets reportedly have done, out of titanium or even significantly stronger steel, and has been slow to upgrade present-day nuclear reactors to get more power for their weight.

"Rickover has been the designer, the developer, the operator and the one who writes the safety rules. We've got to do something different.

Even if those remarks are dismissed as simply anti-Rickover talk, Pentagon executives and the uniformed Navy do agree on these points regarding the status of nuclear submarines:

Intelligence gathering has provided overwhelming evidence recently that the Soviet's Alpha submarine has achieved a speed of 40 knots, compared to a top speed of about 33 knots for the U.S. Navy's Los Angeles-clas high-speed attack boat. The titanium Alpha, which sources said is now being built in quantity, also can dive much deeper than the Los Angeles.

The $1.5 billion cost of the first Trident missile submarine is almost twice the original estimated and must be reduced significantly.

Budget limitations mean the long-time Navy goal of 90 nuclear attack submarines will not be achieved by building only Los Angeles-class boats, which cost $500 million each. A cheaper nuclear attack boat must be included in the mix.

Trident missile submarines are not being built fast enough to match the rate at which the aging Polaris and Poseidon fleet is being retired.

Because there is nothing radically new in either hull material or power plants abailable to those challenging the Rickover philosophy, the Pentagon is considering a series of quick fixes in hopes of solving some of the problems they see in the submarine fleet. The fixes under consideration include th following:

Replacing the reactor Rickover designed for the Trident submarine with one providing half the orsepower to save weight and construction costs. The Pentagon already has spent million assessing this approach and is expected to ask Congress for a big addition in the fiscal 1981 budget now being prepared.

The Navy is working on a small attack sub - about 5,000 tons compared to 7,000 tons for the Los Angeles - that would be constructed from equipment originally designed for the Los Angeles, Narwahl and Sturgeon submarines.

David E. Mann, assistant secretary of the Navy for research, is publicly proclaiming that billions of dollars could be saved if the Navy turned away from Rickover's giant submarines and built smaller missile and attack boats.

But here the civilian leadership of the Pentagon is in disarray. One highranking Defense executive said in an interview that the Navy study of alternative submarine design fails to support with solid data the case that Mann is making in statements to the press.

During all this second-guessing of Rickover's past decision on submarinees, the admiral himself, according to Navy insiders, has been uncharacteristically quiet.

One guess is that he is waiting for the dust to settle. Although Rickover still has clout in Congress, many of his old allies there - especially the leaders of the Joint Atomic Energy Committee - have retired or lost much of their political power.

President Carter is an outspoken admirer of Rickover, but he did veto a defense bill last year because it contained money for another nuclear aircraft carrier, which the president opposed. Rickover had championed the nuclear carrier.

Also, the majority of Congress decided to reject Rickover's advice and pay the contested bills of of shipbuilders as civilian leaders of the Navy had recommended.

Until now, Rickover's philosophy of building submarines has gone without serious, sustained challenge. This seems to be changing.