A leading Senate opponent of the MX missile charged yesterday that the Air Force apparently made a "staggering" miscalculation in estimating the cost of an alternate missile system.

Sen. Mark O. Hatfield (R-Ore.) said the apparent mistake raises serious questions about the cost-effectiveness of the MX mobile missile, which President Carter approved last week for full-scale development.

Hatfield based his charge on a Library of Congress study that he commissioned on the MX issue. The study says the Air Force overestimated the cost of making existing Minuteman III missiles mobile by $10 billion to $12 billion.

If that amount is deducted, the study claims, retrofitting Minuteman III missiles would be less expensive than building the MX, which the adminstration estimates will cost $30 billion.

"The Air Force has constantly and officially repeated that the primary reason for deployment of the MX missile was its relative cost-effectiveness when compared with other options," Hatfield said. "Once the cost-effectiveness of the MX is brought into serious question, the entire rationale for the MX missile collapses like a house of cards."

The MX is a mobile intercontinental ballistic missile that is designed to be less vulnerable to Soviet first-strike attack than the stationary Minuteman. The MX would carry up to 10 warheads.

Hatfield argued that a mobile Minuteman missile could be developed more cheaply than the MX and be "far less strategically destabilizing."

The Library of Congress study said the Air Force envisioned a need for 12,000 protective shelters to house 550 Minuteman III missiles compared to only 4,500 shelters for 200 MX missiles.

"Calculations show that this [assumption] is in error and that the same number of shelters is adequate for both forces," the study said.

However, the study's figures appear to be outdated. Administration officials said last week that the 200 MX missiles would be scattered among 8,000 to 9,000 shelters.

The study also said subtracting the cost for the extra Minuteman shelters would trim $10.2 billion to $12.3 billion from the cost of making the Minuteman mobile. That would make the Minuteman III option between $1.2 billion and $3.3 billion cheaper than developing the MX.

Those savings, however, are based on estimated costs for the MX of $20 billion and for modernizing the Minuteman of $29 billion. The Air Force now estimates the cost of the MX at $30 billion.

Col. Donald Wakefield, a Defense Department spokesman, said the Air Force had not seen the study and would have no comment until it had a chance to analyze its contents.