An international arbitration tribunal in Geneva has used a geometric formula to settle a disagreement between France and Britain over oil-prospecting rights at the western approach to the English Channel.

The binding decision, reached July 18, was made public yesterday in Paris and London. It gives France a bigger portion of the ocean bottom than Britain had been willing to concede, but provides that the continental shelf limit for the British Channel Islands be set 12 miles rather than the 6 miles that France had wanted.

Guy de Lacherriere, head of the French delegation to the tribunal, estimated the protion of the disputed area granted to France at 3,360 square miles.

At issue was the placement of a median line between the two countries. Britain wanted it to be equidistant from the Scilly Islands, off Land's End, westermost point on the English mainland, and Ushant Island off the coast of Brittany in France. France said this would be unfair because a geographical "accident" had placed he Scillies farther west than Ushant.

The tribunal devised a formula that incorporated measurements taken along both the Ushant-Scillies line giving France more area than it would have had if the median had been based on the Scillies line alone.