The city of New Carrollton's plans to annex a future multimillion dollar industrial, hotel and commercial complex as well us two small residential communities were jeopardized yesterday when a judge ordered recomputation of a complex annexation petition.

The ruling by Prince George's Circuit Judge Perry G. Bowen came in a suit filed by five persons, including three businessmen from the developing Metro East complex that the city wants to annex. Bowen decided the suit in favor of the five on Tuesday, saying that New Carrollton's mayor and council violated Maryland's public information law by formulating annexation strategy in closed meetings a year ago.

To penalize the city for breaking the law, Bowen said yesterday he will require recomputation of the petition that set the annexation machinery in motion.

The territory the city wants to annex includes the residential communities of West Lanham and West Lanham Hills, as well as the so-called Metro Triangle, which includes Metro East. The petition drive centered in the communities, and signatures were collected from residents anxious to be included in the city and benefit from its municipal services.

The annexation rules require a successful petition to contain the signatures of at lest 25 percent of the voters in the total area to be annexed plus the owners of real estate assessed at at least 25 percent of the total assessed valuation of the area.

The petition, filed Sept. 28, met both tests. The total assessed valuation of the land proposed for annexation was $11,446,000.

To penalize the city, Bowen ordered the petitions recalculated according to the assessed value of the area last Jan. 13. Since the signatures were collected, the value of the land in the area has presumably risen because of inflation and rapid development of the Metro East complex.

It remains to be seen whether the value of the petition signers' property has risen enough to remain above the 25 percent level necessary to validate the petition drive.

Bronson Row, president of the West Lanham Hills Citizens Association, who led the petition drive last year, said that it may be impossible to collect more signatures if they are needed to bring the assessment total to 25 percent. He noted that the petition already contains signatures of 195 of the 282 homeowners in the area.

The Metro East businessmen who object to annexation claim they will receive no municipal services in return for the additional taxes they would pay if their property were made part of the city. Annexation would give the city an estimated $214,000 in yearly tax revenue by 1983 after development of Metro East is completed.

If the petition is upheld the next stage in the annexation procedure -- a referendum in West Lanham and West Lanham Hills -- will go ahead as scheduled next Saturday.

If it is determined that the petition drive failed, the city can suspend the referendum and appeal Bowen's ruling, seek additional petition signatures over a 30-day period or abandon its annexation plans.