Secretary of State Alexander M. Haig Jr. said yesterday that "extensive studies" have been completed within the government about ways to thwart what he charged are increasing Cuban subversion and terrorism in the Western Hemisphere.

Haig declined to disclose the nature of the U.S. plans, which he said are under review before being presented to President Reagan for approval.

Speaking to a foreign policy conference with out-of-town journalists at the State Department, Haig indicated that the aim will be, as in the 1960s, "to make the risks to Cuba seem to be more costly than the advantages" of intervening in Latin America.

Speaking to the same conference, Haig's senior deputy for inter-American affairs, Assistant Secretary of State Thomas O. Enders, said, "Our policy toward Cuba is under very active consideration. You haven't heard the last of this at all."

Enders disclosed that the Reagan administration has begun to tighten the economic embargo against Cuba, and other officials said additional measures on the longstanding embargo are contemplated.

The measures already taken, according to State Department officials, include the designation so far this year of 18 firms in the United States, Panama and Jamaica as fronts that are intended to circumvent the embargo on trade with Cuba. After such a designation, it becomes illegal for Americans to deal with them because of assets control regulations.

Cuban diplomats in Washington, meanwhile, released a statement denying recent reports that 500 to 600 Cuban special troops were flown to Nicaragua last month.

The statement, released through the Cuban interest section of the Czechoslovak embassy, called the reports "an absolute lie devised from A to Z by Yankee imperialism." The statement said Haig had passed the reports along to other governments after they were published in the U.S. press.

Enders, speaking to the policy conference, expressed sharp concern about the political direction of Nicaragua's leaders, saying, "It looks very much as if they are headed toward a repressive, totalitarian system."

He said the administration has "tried very hard" to reach accommodations with the Sandinista government, including "very frank talks" he conducted in Managua last month. At that time the United States made a number of proposals to Nicaraguan authorities, Enders said, but there has been no response.

Enders charged that Nicaragua is engaged in an extensive Soviet weapons buildup, saying that the country has acquired Soviet T55 tanks from Algeria and Libya and that Nicaraguan pilots are being trained to fly Soviet Mig fighter planes.

He said Nicaraguan military forces are already three times as large as those that existed under former president Anastasio Somoza. A possible effect, according to Enders, would be to frighten conservative regimes in the area.

Haig, in other comments to the State Department conference, said:

* He and Reagan have "drawn some encouragement" from Haig's lengthy meetings in New York last month with Soviet Foreign Minister Andrei F. Gromyko. Speaking of the planned meeting with Gromyko in January in Geneva, Haig said a continuation of the "same level of give and take" would provide hope of resolving some Soviet-American differences.

* He did not discuss the possible sale of U.S. arms to China or Taiwan during the first day of Washington meetings with Chinese Foreign Minister Huang Hua. Huang met Haig, Reagan and Secretary of Defense Caspar W. Weinberger yesterday, and is to continue his meetings with top American officials today.

* The demonstrations in Western Europe against American nuclear weapons are not part of a trend toward neutralism but "a unique convergence of concerns about all things nuclear." He called for "a keen sensitivity" for the nuclear concerns of Europeans to be exercised in American public statements.