Worries about a second submarine gap opening up, this one killer subs rather than missile boats, has prompted the sharpest senate challenge yet to Navy undersea warfare plans.

The Senate Armed Services Committee is demanding that the Navy think smaller and cheaper when it comes to building submarines from now on rather than insist on getting the biggest and best money can buy.

Otherwise, warns the committee, the Navy is going to price itself out of new missile and killer subs the way the Air Force priced itself out of its new B1 bomber.

The committee, which the Navy cannot safely ignore, followed up its warnings with a specific request to consider novel ideas for submarines including nonnuclear power plants.

Congressional and Navy sources said the Senate has never previously gone so far in challenging the submarine service generally and, by implication, Adm. H. G. Rickover, its ironhanded director of nuclear propulsion.

Adm. James L. Holloway III, chief of naval operations, which insisting he could see no alternative to Trident, acknowledged in testimony just released by the committee that the Navy indeed faces a gap in submarine missiles because the existing Polaris-Poseidon fleet will wear out before enough Tridents are built to take their places.

Under the terms of the previously negotiated strategic arms limitation treaty, Holloway said in Senate testimony released last week, "We will be below the number of sea-based missile tubes that would be appropriate under the overall level."

In contrast to the 656 submarine missile tubes available today from the fleet of 41 Polaris and Poseidon subs, the Navy estimated it will drop down to 312 submarine missile launchers in 1986 unless ways are found to keep Poseidon subs on duty longer than their scheduled 20 years.

"We are looking very hard at what we can do to extend the service life of Poseidon," Holloway said. "We are convinced the hull is sound. We have got reservations regarding the propulsion plant. Initially, we didn't plan to operate those nuclear reactors and supporting systems for more than 20 years . . ."

Sen. Jake Garn (R-Utah) after noting that the B1 bomber had priced itself out of existence, asked Holloway: "Are we going to price ourselves out of the market on Trident?" The first Trident, now under construction in Groton, Conn., is five feet longer than the Washington Monument is high and is expected to cost about $1.5 billion, counting its 24 missiles.

"I don't see any alternative to the Trident submarine and Trident I missile," Holloway replied. But the Senate Armed Services Committee, in rewriting President Carter's fiscal 1979 defense program, took a contrary view.

The committee added $36 million to the president's budget so the Navy could "investigate the feasibility of replacing some of the Polaris/Poseidon" submarines with a missile sub "which would have a lower initial acquisition cost than the Trident."

Swinging its guns to the Navy's attack submarine program - the 688 class of killer subs which General Dynamics recently threatened to stop building until back bills were paid - the committee said economies are needed here, too, to avoid a gap.

"If the submarine force level is to be maintained," the committee lectured the Navy, "it would appear necessary to adopt a high-low" mix approach to the attack submarine force in which SSN-688 class submarine are complemented in some missions by less capable, less expensive submarines." The Navy estimates each of its planned 37 attack subs in the 688 class will cost $257.6 million.

"Therefore," said the committee in its report, "the committee requests the Navy, in cooperation with the office of secretary of defense, to prepare and submit to the Congress no later than Feb. 1, 1979, a comprehensive study and evaluation of all options and alternatives for maintaining an attack submarine force level of not less than 90 submarines.

"This study," the committee directed "shall include evaluations of lower cost, lower capability nuclear propelled submarines and the use of nonnuclear submarines or some missions, in combination with submarines of the SSN-688 class."

Although this might sound to outsiders like just another logical request of a congressional committee, Navy critics of Rickover jubiliantly greeted the Senate instructions as a major breakthrough. Finally, they said, the Navy will be forced to look at alternatives to wrapping submarines around Rickover's progressively larger nuclear power plants.

One undersea warfare specialist said: "With Holloway retiring and Congress demanding at least a look at nonnuclear power for subs, it's a whole new ball game."