A recent decision by Britain's highest court denying an American company access to information from a British company points up the vexing problems arising out of the American view that its antitrust laws apply outside its national boundaries.

The ruling also highlights the thorny issues inherent in the clash of the economic system, legal system, principles of law and foreign policy of one country with another in a world that seems to grow smaller every day.

The ruling, by the "Law Lords," five lawyers of the House of Lords who sit as Britain's highest court of appeal, denied Westinghouse Electric Corp's bid to get information from Rio Tinto-Zinc Corp. and seven of its officials for use in a U.S. court proceeding pitting major American utilities against the U.S. giant.

The case before the lords was a two-part appeal, one involving documents and one involving oral testimony. The court of appeal had ruled that Westinghouse could take oral evidence from the RTZ officials but couldn't have access to documents from RTZ files. RTZ had claimed "privilege" for the documents under British law on the grounds that the documents could incriminate the company and expose it to penalties under competition laws of the European Economic Community to which Britain belongs. (Companies not individuals are subject to those laws.)

Westinghouse had sought documents from the files of the British-based major uranium producer and oral testimony from its executives in a quest for evidence to support its contention that the activities of a worldwide uranium cartel, to which RTZ belonged, contributed to a substantial increase in the price of uranium and made it difficult for Westinghouse to secure supplies. Westinghouse is now on trial in Richmond, Va., for backing out of long-term uranium supply contracts with a dozen major utilities, including the Virginia Electric & Power Co.

The oral and documentary evidence from RTZ was sought by the U.S. District Court in Richmond under the terms of a Hague Convention, which the United States and the United Kingdom have signed, allowing testimony to be taken from citizens of other countries in civil proceedings.

Last week, the "Law Lords" unanimously ruled that the American court's request failed to comply with British legal requirements. Lord Dilhorne, speaking for the lords, said the U.S. court's request of Oct. 21, 1976, probably was "regarded as a step in the normal process of discovery in American courts" but constituted "what we would call a 'fishing expedition.'"

In enforcing the Hague Convention, the home country generally has to be enforcing norms consistent with its own domestic law, a Justice Department official here explained. He said the British take the view that broad subpoenas such as those sought by Westinghouse cannot be granted under English common law.

RTZ lawyers did not deny that information Westinghouse might obtain in Britain might be relevant in its U.S. case, but they contended that was not the issue. British law doesn't allow the exercise "even if in the course of the 'fishing expedition' you may catch a fish." RTZ's barrister Kenneth Rokison argued before the lords.

More importantly, the lords took issue with the U.S. government's involvement in the case, underscoring the growing foreign concern about American persistence in seeking to apply its antitrust laws and principles of competition outside its national boundaries.

The lords determined that the actions of the Justice Department to grant the RTZ officials immunity from criminal prosecution under U.S. antitrust laws represented a misuse of the convention and the process set up to execute it. The Justice Department had granted the RTZ officials immunity following an unsuccessful attempt by Westinghouse lawyers to question them in London during the summer. When testimony began, the officials took the Fifth Amendment, noting that a grand jury had been impaneled in Washington to investigate the cartel activities and any information they might give could be used to indict them.

In granting the RTZ officials immunity, the Justice Department had acknowledged that such a step was unusual - immunity in a civil case was only granted once before and the situation was somewhat different - but suggested that "extraordinary circumstances" made the grant in the public interest.

Justice officials admitted that the government was having difficulty getting information in its grand jury investigation of the uranium cartel and wanted to know what the RTZ officials would have to say.

The immunity grant, however, prompted the intervention of the British government in the case, too. During a hearing before the lords in October, Samuel Silkin, Britain's attorney general, agreed with RTZ that the United States "intervention" transformed the nature of the original request by a U.S. court for the assistance of the British courts in a civil proceeding into something different.

Silkin complained, as the British had in a memo sent to the U.S. government earlier through diplomatic channels, that the United States was violating British sovereignty, seeking to extend its laws into the territory of others.

The British government did not agree with the U.S. position that it had the authority to investigate non-U.S. companies and non-U.S. individuals with respect to their actions outside the United States, he told the court.

"For the purposes of U.K. sovereignty, the U.K. does not recognize any such investigation as having any validity or as being proper," Silkin said. "Any use of the U.S. antitrust laws or procedures . . . except with the authority of the U.K. is an invasion of and prejudicial to U.K. sovereignty."

The lords unanimously agreed. "The evidence is sought for the purpose of an antitrust investigation into the activities of companies not subject to the jurisdiction of the U.S.," Lord Wilber force, the presiding law lord said. "It is axiomatic that, in antitrust matters, the policy of one state may be to defend what it is the policy of another state to attack."

Neither Westinghouse lawyers nor Justice officials were particularly surprised by the outcome. Though a little disappointed, attorneys for Westinghouse said that had they won in the British court they expected to use the testimony from RTZ and any documents they might have gotten more to "fill in gaps" than anything else.

Attorneys William R. Jentes and Samuel A. Haubold both likened the story of the uranium cartel to a giant puzzle they had already been able to put together fairly well, each new piece of information just filled a small hole left open on the board.

Justice officials, too, had not been optimistic about the lords' decision. They knew the wording of the grant of immunity - stressing the testimony's importance to the grand jury investigation and not Westinghouse's Richmond case - was made much of in the British court by RTZ lawyers and the British government. The document's wording was the price paid to clarify the department's own policies on the unusual matter, one official explained.

The U.S. position was that the grant of immunity wasn't in "the nature of an intervention," according to Douglas E. Rosenthal, chief of the foreign commerce section of the Antitrust Division.

It was not a "misuse of the process" if there happened to be "collateral or ancillary" interests, he said. The primary purpose of the immunity grant was to let Westinghouse go forward with its case under the terms of the convention.

Rosenthal and others see the RTZ case as part of a broader problem that promises to come up again, and probably more frequently as international trade matters and the operations of multinational corporations force more confrontations between countries and their companies.

Rosenthal poses the dilemma: "What do you do when two jurisdictions with legitimate claims to jurisdiction over particular conduct and particular persons have diametrically opposite views of how that conduct ought to be viewed?"

The U.S. position has been that the it has authority over acts contravening U.S. antitrust laws even if they are committed abroad and by foreign nationals, if those acts have economic effects in the United States.

For instance, if the actions of the uranium cartel, whose members included companies in Britain, Canada, France, Australia and South Africa, forced the cost of uranium up in the United States or made it difficult for Westinghouse to secure supplies - two claims now at issue - then the actions should come under the jurisdiction of U.S. antitrust laws.

This is not a view generally subscribed to by other countries. Few nations have any regulations resembling U.S. antitrust laws; most countries don't particularly agree with the general economic principles from which they stem. It was not surprising that when Silkin presented the British government position to the lords, he noted that three nations - Australia, Canada and France - had specifically asked to be associated with the British position.

The U.S. has recently begun to acknowledge the tensions its views have created among many of its long-standing allies. It has adopted what appears to be a more conciliatory approach in its pursuit of U.S. objectives when U.S. antitrust enforcement appears to conflict with the laws and foreign policies of others.

Associate Attorney General Michael J. Egan told the International Bar Association last month that the Justice Department would make a concerted effort to consult formally with interested foreign governments to explore means of accomodating conflicting national interests.

He also said the department has decided to notify a foreign government when an Antitrust Division official wishes to conduct investigative interviews or other official business within its territory.

"Without foreign cooperation, the department lacks the means to obtain critical evidence from abroad and lacks the ability to obtain effective relief as to foreign conspiracies involving foreign products," Egan said.

In addition to the continuing grand jury investigation into the worldwide uranium cartel, the Justice Department is carrying out a criminal probe into shipping conferences in the North Atlantic, and civil investigations of the international oil, platinum, and zinc industries as well as an investigation of restrictions in certain technology licensing agreements against sales by foreign licenses in the U.S. market.