Ranking European, Japanese and American diplomats continued exchanges yesterday on a formula for East-West trade relations, and a new European position reportedly has been presented to U.S. officials.

"We are still consulting," Canadian Ambassador Alan Gotlieb said after the morning meeting at the State Department. "I don't want to make predictions about optimism or pessimism. We're into some real important consultations and we feel we should be able to get somewhere with them."

The talks are designed to reach agreement on trade with the Soviet Union covering such issues as credits, energy dependence and sale of high-technology items with possible military application.

The Reagan administration has said that it would lift the gas pipeline sanctions that have affected several European companies if an alternative could be reached. Officials have pointed to agreement on an East-West trade policy as such an alternative.

Yesterday's meeting was the fifth set of high-level talks in two weeks on the issue, which has sharply divided Europe and the United States.

State Department spokesman John Hughes said that no further meetings have been scheduled but stressed that this did not rule one out.

A European official called yesterday's session "useful" and said "we can confidently expect to meet again in the near future."

Symptomatic of the difficulty in reaching an accord was the statement by French President Francois Mitterrand yesterday in Paris following talks with British Prime Minister Margaret Thatcher.

"The sanctions are unilateral, unjust and legally questionable. If things can be worked out, so much the better . . . . There can be no question of negotiating something that is not negotiable," Mitterrand said.

This reflected deeply held French feelings over the sanctions and the difficulty facing the Washington negotiators as they try to reach an agreement that will satisfy the administration's demand for alternatives yet not appear to be linked to the pipeline issue.