The Reagan administration, charging that the International Court of Justice is being misused by Nicaragua and the communist bloc for political and propaganda purposes, yesterday ended the 39-year-old policy under which the United States had agreed to abide by the court's decisions automatically.

The move followed a U.S. decision in January to boycott court proceedings on charges that the United States has supported guerrillas trying to overthrow Nicaragua's leftist government. The walkout came after the Hague-based tribunal, commonly called the World Court, rejected U.S. arguments that it lacked jurisdiction in the case.

The latest administration action reversed almost four decades of U.S. efforts to obtain widespread international recognition of the World Court as a forum for resolving disputes through international law.

The administration contended yesterday that the United States had been "disappointed" in its original expectations that other countries would follow the U.S. lead and voluntarily accept the court's jurisdiction.

Instead, the administration noted in a State Department statement, fewer than one-third of the world's nations have accepted the court's jurisdiction, "and the Soviet Union and its allies have never been among them."

The U.S. action now means that only Britain, among members of the U.N. Security Council, automatically abides by court rulings.

However, the action seemed likely to provoke a new wave of criticism that it is undermining respect for the rule of law and fueling suspicions in other countries that the United States is trying to cover up its activities against Nicaragua.

Sen. Mark O. Hatfield (R-Ore.) last night said, " Ayatollah Ruhollah Khomeini of Iran , Libyan leader Muammar Qaddafi and all the other world-class thugs who thrive on the rule of the jungle will no doubt welcome this decision."

Hatfield said he was introducing a non-binding resolution that would put the Senate on record as "deploring" the administration action, and a similar resolution was expected in the House.

What the administration did yesterday was to scrap its 1946 agreement to comply automatically with the court's rulings in cases involving other nations that also had accepted the tribunal's "optional compulsory jurisdiction." That means that countries accepting the court's blanket jurisdiction can sue each other.

U.S. officials stressed, however, that the United States will continue to deal with the court in cases in which Washington has agreed with other governments, either through treaties or other agreements, that the court is the proper forum for resolution of their disputes. Typically, such cases involve commercial, legal or boundary problems.

There is one American, Stephen M. Schwebel, among the court's 15 judges, who are elected by the U.N. General Assembly and Security Council. Abraham D. Sofaer, the State Department's legal advisor, said Schwebel's position will not be affected by the U.S. action. Sofaer also noted that the United States will continue paying its U.N.-assessed share of the court's administrative costs.

In defending the administration's position that the Nicaraguan charges do not belong in the World Court, Sofaer contended that the matter involved a political dispute that more properly should have been brought to a political forum. He suggested more appropriate venues would be the Security Council or the Contadora negotiating process, which is seeking a comprehensive peace plan for Central America.

"We believe that the Nicaraguans are supported by Cuba and the Soviet Union," Sofaer said. "We believe that the Nicaraguan effort to export revolution is a part of a general effort by communist regimes to take over and undermine democracies."

Under those circumstances, he insisted, the United States and its allies have the right to engage in "collective self-defense" against Nicaraguan activities.

When the administration decided in January to boycott the Nicaraguan case, there appeared to be general agreement among international law experts that the United States had ample precedent for refusing the court's authority since so many other countries have rejected unlimited court jurisidiction.

However, administration sources privately acknowledged that they had encountered sharp differences within the legal community about the wisdom of acting solely on that legal point. Many experts have argued publicly that the United States would have been wiser to allow the case to be argued on its merits.

But, the sources said, the administration decided in January that further hearings would expose the degree to which the United States had aided the Nicaraguan guerrillas. They also said such disclosure would give Nicaragua a major propaganda victory and potentially damage U.S. intelligence activities.