The Reagan administration said yesterday it is investigating allegations that a U.S. computer system that could help determine targets for nuclear missiles had been illegally shipped to the Soviet Union.

According to a published report, a Los Angeles firm, Essex Marketing Co., is under investigation for an alleged shipment Dec. 21 of a computer capable of simulating nuclear explosions to an engineering college in Zagreb, Yugoslavia.

However, the computer was diverted and wound up in the Soviet Union instead, shipped to Moscow via Belgrade, the Yugoslav capital, and East Berlin, it was reported.

"We're concerned by this apparent diversion and are working with Yugoslav authorities to determine exactly what happened in the specific case," said State Department spokeswoman Phyllis Oakley.

She said the matter is "under investigation by the U.S. Customs Service and the Commerce Department."

U.S. law prohibits export of items with possible military application to Soviet-bloc nations. Other high-technology goods may be shipped to those nations only with approval of the Paris-based Coordinating Committee for Multilateral Export Controls, COCOM. Yugoslavia, a communist nation, is not officially aligned with the Soviet Union. High-technology shipments there do not require COCOM approval.

Donald Creed, a Commerce Department spokesman, said government investigators had uncovered "an elaborate scheme to divert a high-technology computer." He said it was the first "diversion that we are aware of concerning Yugoslavia."

He declined to discuss details on the grounds that the investigation is still under way.

The Washington Times in yesterday's editions said the computer, made by Floating Point Systems of Beaverton, Ore., can be used by military planners to gauge the impact of nuclear explosions and to help determine optimal targets for nuclear missiles.

The newspaper said Floating Point was not a target of the investigation. It quoted Essex president Walter Patzl as denying knowingly selling anything illegal to Warsaw Pact nations.