Deputy Secretary of State John C. Whitehead said yesterday that he "fervently" hopes that the United States will not have to take military action against Libya but warned that the answer lies not in U.S. hands but in those of Libyan leader Muammar Qaddafi.

Whitehead, who returned Friday from a 10-day tour of Canada and Western Europe to explain U.S. economic sanctions against Libya for alleged support of terrorism, caused a stir when he stressed to reporters that President Reagan has "reserved the right to come back to the military option in case the nonmilitary, peaceful measures don't work."

Whitehead's stress on that point appeared to run counter to administration efforts for two weeks to play down talk of retaliatory strikes against Libya for its alleged complicity in terrorist attacks at the Rome and Vienna airports Dec. 27.

"We all can hope and pray, and certainly our allies told me that they hoped and prayed that we would not need to resort to the military option," he said. ". . . But the answer lies . . . in Qaddafi's hands because, if he continues to sponsor and direct and conduct these atrocities . . . we cannot stand idly by."

Whitehead said he regards as "incontrovertible" the information that he presented to the allies to back U.S. charges that the Libyan-supported Abu Nidal terrorist group carried out the airport attacks.

Other State Department officials, speaking on condition that they not be identified, have said Whitehead's evidence made a strong circumstantial case against Libya but did not contain "a smoking gun."

Whitehead insisted that the evidence "did go beyond the circumstantial. It was quite specific." He added that most had been received by U.S. intelligence sources after Jan. 6 when Reagan announced the U.S. economic measures against Libya.