The Reagan administration has formally notified the NATO alliance that its European members, not the United States, will have to pay the costs of moving the Air Force F16 fighter planes from Spain to a new home in Italy or another European country, U.S. officials said yesterday.

Administration officials acknowledged it was unusual to make such a direct demand to NATO partners but added that Congress gave them no choice under last year's budget laws, which forbade the United States from paying such support costs for defending Europe.

Alton G. Keel Jr., U.S. ambassador to NATO, informed alliance officials of this situation earlier this week, officials said, and received a sympathetic hearing but no offer of money to pay for the relocation of the 72 fighters at the Torrejon base near Madrid.

While Italy has discussed the possibility of providing a site for the planes, Defense Secretary Frank C. Carlucci said in Lisbon yesterday that no agreement has been reached nor is one likely in discussions starting in Rome today.

Although the administration has put no price tag on relocation costs NATO would have to absorb, one official said the total is bound to be "in the hundreds of millions," given probable needs to lengthen runways and build houses for the 4,500 members of the 401st Tactical Air Wing.

"I myself would like to see it remain in Europe," Carlucci said of the 401st, "but that's a NATO problem."