MANCHESTER, N.H., FEB. 15 -- Republican presidential candidate Pat Robertson spent his last day of the New Hampshire primary campaign fending off the fallout from his allegation at Sunday's Republican debate that the Soviet Union has put intermediate-range nuclear missiles in Cuba.

The White House categorically denied the charge today as Robertson tried to back away from his own "Cuban missile crisis," which arose during the League of Women Voters debate here when he said that the Soviets "have put SS4s and 5s in Cuba." He said it would be "outrageous" to call the statement a "gaffe," contending that he said only that there "may be" missiles in Cuba and that "I have no idea" whether they contain nuclear warheads.

"All I did was ask the gentlemen, 'Is this true?' " Robertson said. "I mean, I didn't come through bombastic." The statement drew startled reactions from his fellow Republicans on the stage Sunday and prompted a deluge of phone calls to the White House, including one from the campaign staff of Vice President Bush, who asked for a formal disavowal.

"The SS4 and SS5 missile sites established in Cuba between late August and mid-October 1962 {the time of the Cuban Missile Crisis} were confirmed as having been destroyed in that year," the White House said in a statement issued by presidential spokesman Marlin Fitzwater in Santa Barbara, where President Reagan is vacationing at his mountaintop ranch.

"We also are confident that the missiles were removed from Cuba at that time," the statement said. "We have had extensive intelligence collection directed at Cuba since 1962 and have no evidence that SS4, SS5 or other strategic missiles have been deployed there."

Touring a gun factory in Rochester and displaying his pugilistic prowess at a Manchester gym, Robertson was dogged by questions about the source of his assertion. Robertson eventually identified his "only source" as David S. Sullivan, a senior Republican staff member of the Senate Foreign Relations Committee.

Sullivan was quoted by NBC News as saying "we do not know" whether there are Soviet missiles in Cuba.

In Washington, Sullivan refused to say whether he gave Robertson the information on which the candidate based his debate assertion, but he said it was contained in a Feb. 9 speech by Sen. Jesse Helms (R-N.C.) in Concord.

Robertson appeared irritated by continued questions about the flap, and at the Thompson Center Arms factory in Rochester, he picked up a .50-cal. Hawken muzzle-loaded rifle and, with a grin on his face, pointed it at journalists.

"We like NBC; you don't understand," the candidate quipped as he aimed the rifle at a camera crew from that network.

At a forum in Dover, audience reaction was hostile when Robertson said he would have "knocked down" the Berlin Wall had he been president at the time it was built, and told his audience, "Folks, if war is thrust upon us, there's only one way to deal with it. Get in, win it and get out."

"I'm standing here really afraid," Kent Evans of Portsmouth, a supporter of Sen. Robert J. Dole (R-Kan.), told Robertson. "When you start talking about using war the way you just talked, of taking tanks against the Berlin Wall, I'm shaking."

This was greeted by a burst of applause.

Staff writers Lou Cannon in Santa Barbara and David B. Ottaway in Washington contributed to this report.