A federal appeals court panel, in a victory for conservationists that complicates U.S. relations with Japan, ruled yesterday that the United States is required to reduce Japanese fishing rights in U.S. waters because Japan refuses to abide by an international ban on commercial whaling.

In a 2-to-1 ruling, a panel of the U.S. Court of Appeals for the District of Columbia said federal law requires the Commerce Department to impose the economic sanctions, which the Reagan administration had sought to avoid through a separate agreement that would have allowed Japan to continue taking whales for two years longer than allowed by the International Whaling Commission.

The decision upholds a ruling here in March by U.S. District Court Judge Charles R. Richey.

The appeals panel delayed the ruling for 90 days, to allow time for an appeal. A spokesman for the Japanese fishing industry said the case would probably be appealed either to the full Court of Appeals or to the Supreme Court.

The ruling requires the United States to halve the Japanese fishing quota, which was worth about $500 million last year.

In April, the Japanese government announced that it would end all commercial whaling by 1988, but only if appeals courts in the United States overturned Richey's ruling imposing the immediate sanctions.

Yesterday a Commerce Department spokesman said department officials "have committed ourselves to defending this agreement," but said a final decision had not been made on whether the United States will participate in an appeal.

A Japanese embassy official expressed "disappointment" over the decision, but declined further comment. However, Alan Macnow, a spokesman for the Japanese fishing and whaling industries, said the decision creates "a lot of risk" for the United States because Japan purchased about $600 million in U.S. fish products last year, about four times more than Americans bought from Japan.

"The Japanese are very angry that America is just hitting Japan over the head with a club about this," Macnow said, "so conceivably the Japanese fishing industry and consumers could stop buying American fish. It's one of the few areas where the U.S. enjoys a favorable balance of trade."

Russell Wild of Greenpeace, one of the 12 environmental and animal rights groups that filed the lawsuit, said the appeals panel's decision was a "tremendous victory" which would be seen as "one of the most significant events in stopping the slaughter of whales."

In yesterday's decision, Judge J. Skelly Wright declared that under two federal laws, the Pelly Amendment of 1971 and the Packwood-Magnuson Amendment of 1979, Commerce Secretary Malcolm Baldrige is required to certify that by continuing to catch whales, Japan is "diminishing the effectiveness" of an international fish conservation program, and thus "automatically" trigger the sanctions.

"Specially," Wright said, "where a foreign country's nationals harvest whales in excess of commission quotas, certification is mandatory and nondiscretionary."

The opinion was also signed by Judge Edward A. Tamm.

In a dissent, U.S. District Court Judge Louis F. Oberdorfer said it was unclear that the statutes gave the commerce secretary no discretion in the matter. He added that the court order "entangled the judiciary in foreign policy" and required the U.S. government to break "its word" that sanctions would not be invoked if Japanese whaling stopped in 1988.

The commission voted in 1982 to declare a worldwide ban on all commercial whaling, starting in 1986. However, countries were able to exempt themselves under terms of the treaty by filing formal objections, which Japan did. But U.S. law is not as flexible as that treaty, the court said, in effect.

Japan also objected to an IWC ban on catching sperm whales, which went into effect this year.

The Japanese caught 400 sperm whales in the northwest Pacific this year out of an estimated population of 210,000 and 1,980 minke whales off Antarctica out of a population of 250,000, Macnow said.