The White House, in an unusual move, has asked the Senate to reconsider its confirmation of retired Adm. Elmo R. Zumwalt to an arms control advisory board, bowing to the wishes of Virginia's two senators, both longtime political and personal adversaries of the outspoken former chief of naval operations.

Zumwalt, a Democrat who broke with his party to support Reagan in 1980, was one of 14 appointees to the General Advisory Committee of the Arms Control and Disarmanent Agency. All were confirmed by a voice vote in the Senate Foreign Relations Committee last Tuesday and by the full Senate without debate on the following day.

Zumwalt's nomination, however, cleared Congress without catching the eye of either Sen. Harry F. Byrd Jr. (I-Va.) or Sen. John W. Warner (R-Va.) Considering that Zumwalt, a McLean resident, ran against Byrd as a Democrat in 1976 and in his memoirs blasted Warner's performance as secretary of the Navy, it was an oversight both men quickly regretted.

"The Virginia senators feel there should be more time and consideration on this matter," Irene Forde, a Warner aide, said yesterday. She would not elaborate on the need for a second vote, except to say that typically a senator is given advance notice by the White House of any home-state nominees. "In this case, it fell through the cracks," she said.

A White House spokesman said the Reagan administration agreed to resubmit Zumwalt's name last Friday as a gesture of "courtesy" to the Senate. According to a spokesman for the Arms Control and Disarmament Agency, the Zumwalt nomination, the only one of the 14 to be resubmitted, is still under review.

"We have been contacted by Sen. Warner on this issue and are in the midst of consultations with the White House with a view to its resolution," said Joseph Lehman, an agency spokesman.

The Virginia senators' sudden interest in an otherwise routine appointment was traced yesterday to Zumwalt's harsh criticisms of Warner in his 1976 autobiography and to his campaign against Byrd the same year.

"Zumwalt has said some nasty things about Byrd and he's said some nasty things about Warner. I expect both gentlemen want a chance to set the record straight," said one Senate aide.

"Those two senators apparently hadn't realized he was on the advisory committee list and when they did, they asked to have a chance to talk about it."

In his book "On Watch," Zumwalt described Warner, who served as secretary of the Navy from 1972 to 1974, as an indecisive administrator "who bent with every political breeze that blew," particularly on decisions involving defense contracts in key congressional districts.

He also criticized his former boss for his handling of racial tensions in the Navy.

In 1978, when Warner ran for the Senate from Virginia, Zumwalt called the the Republican "a dilettante" who suffered from a "chronic inability to make decisions."

Like Warner, Zumwalt was a political unknown in Virginia when he announced he would run for the Senate. As predicted, Byrd -- who labeled his challenger as a "carpetbagger" before the campaign got under way -- handily defeated the California-born admiral.

Zumwalt, who now works for Systems Planning Corp. in Arlington, could not be reached for comment yesterday. His former campaign manager, Timonty Finchem, said yesterday he was "mildly surprised" that Virginia's senior senator would take any notice of Zumwalt's appointment, since he never challenged Zumwalt's stands on the issues during the campaign.

"I would question what grounds they have for opposing the nomination except strictly political ones," said Finchem, "I think it would be fair to say that Byrd never mentioned Zumwalt's name once the campaign began or his stand on any foreign policy or defense issues."

"I guess both senators have personal reasons in this. Maybe they got together and decided to take out their irritations on the good admiral this way," Finchem said.

Jack Brooks, Byrd's legislative assistant, said the senator had been caught off guard by the Zumwalt nomination. "We're not angry or anything," said Brooks. "It's just that that advance notice from the White House didn't occur."

Brooks would not predict yesterday what the two Virginia senators will do when Zumwalt's nomination returns to the floor sometime this week. "They haven't indicated what action they will take," he said.

Under Senate procedures, Byrd or Warner could request that the nomination be held -- a request that, coming this late in the session, would be tantamount to its defeat, one aide said.

Or the senators could simply put their objections to Zumwalt on the record and allow the confirmation.