JAPANESE OIL COMPANIES have begun to tell Iran that they won't pay the latest price increases. In response, Iran is not cutting off deliveries to them. That constitutes a victory for the Carter policy of isolating Iran. But there's more to it than that. This incident also demonstrates the limits to Iran's diminishing power in the oil markets.

When a government is overthrown and new people take over, they usually begin testing their new power to see how far it will reach. That's been going on for the past year in Tehran. Japan, with all its handwringing about its total dependence on foreign oil, has established itself in the producers' eyes as the most vulnerable of victims. Last spring, as Iran began moving its prices sharply higher than the others in the Persian Gulf, some of its customers switched to other sources. But Japan anxiously kept paying Iran those premium prices as insurance, hoping to buy itself continued access to Iranian production in the event of another world shortage.

Iran's production kept falling, and Iran's prices kept rising. By this month, Japan was paying Iran $32.50 a barrel for oil that Saudi Arabia was selling at $26, and Iraq at $28. When the Iranians turned the screw again and demanded another $2.50 a barrel, the Japanese finally decided -- insurance or not -- to draw the line. That economic decision coincided conveniently with the latest American requests to Tokyo to break off all commercial relations with Iran.

Meanwhile, Iran's oil has become a less important factor in the world than it used to be. Before the revolution 18 months ago, Iran exported nearly 5.5 million barrels a day. Now the Iranian oil fields are run by people chosen for revolutionary purity rather than technical skill, and exports have been falling sharply. They are now down to a million barrels a day or less -- perhaps one-sixth of the 1978 level.

At that severly reduced volume, the twists and turns of Iranian oil policy will make less difference than they once did. World markets can probably absorb any further Iranian reductions without major strain. It's true that some OPEC countries, like Kuwait, have been cutting back production both to conserve and to keep prices high. But other countries -- Iraq within OPEC, Britain and Mexico outside it -- have been rapidly increasing their production.

The world's oil markets are still tight. The balance between supply and demand continues to be unstable and very nervous. But Iran has now lost its biggest customer, finally demonstrating that there is a point beyond which even the most rapacious of sellers cannot push the most desperate of buyers.