"It's a game of nerves to see who'll crack first -- the Iranians or us," said a British oil man talking about negotiating 1980 long-term contract prices for Iran's oil.

Less than a week before the new year, Iran is not thought to have signed a single contract for its production, which has been running at about 3.5 million barrels a day.

Iran's opening offer was $35 a barrel for essentially the same quality crude oil that Saudi Arabia is offering at $11 less. The major Western oil companies called their representatives home from Tehran when the Iranians acted as if there were no give in their price, industry sources said.

"Shell and British Petroleum will hold out," said the Briton with a barely concealed tinge of national pride, "but I'm worried about the Japanese. They're the weak link. They've been very nervous indeed ever since the majors had to cut back Japan's share because Iran cut them down."

Proof came yesterday that the Iranians understand as well as anybody who is the weak partner in the de facto boycott of Iranian oil that has developed in the past two weeks.

Iran is reported to have offered Japan the same amount of oil for next year as it got this year but at a price of $30 a barrel -- still $6 more than Saudi Arabia and $1.50 more than Iran's own officially announced selling price after the inconclusive meeting in Caracas last week of the Organization of Petroleum Exporting Countries.

Saudi Petroleum Minister Sheik Ahmed Zaki Yamani said upon emerging from a meeting here yesterday with French President Valery Giscard d'Estaing that his country would maintain a price of $24 a barrel rather than go up to a higher compromise price.

Iran's inability right now to dictate its conditions to the world oil market suggests to many petroleum traders that a glut that could drop prices may be developing.

The traders in the business of marrying up the odds and ends of supply and demand known as the spot market handle 3 to 5 percent of the market that is not normally tied up in long-term contracts. The willingness of buyers to pay premium prices for that extra little bit that makes the difference between satisfying their regular customers or not is what explains the phenomenal price levels cited in headlines in recent months.

The spot traders have known for some time that something had to give. Several large cargoes of Libyan crude oil were offered in the two weeks before the OPEC meeting at $45 a barrel. When there were no takers, the Libyans cut the price to $40 and were still unable to find buyers.

A Rotterdam oil trader said in a telephone interview that some heating oil has been selling at $365 a ton this week, but that this was only $10 more than last week -- far less than he had expected it to go up. It is the only item that has been selling at all in the past few days, he said.

He is one of the few spot-market traders who actually operate out of Rotterdam, the center of oil refining for northwestern Europe. "The Rotterdam market," as it is known, is nothing but a network of telephone and telex links among several hundred traders of extra barrels in London, New York, Milan, Singapore and elsewhere. "It's not geographically defined, it's a world market," said an editor of Platt's Oilgram, the bible of the spot market.

Western European governments, notably the French waged a largely fruitless campaign against speculation on the spot market earlier this year.

But the French privately admit that it turned out to be a lot harder to control than they had thought and that the market has, in any case, changed its character.

It is no longer dominated by private traders buying up quantities at spectacular prices in hopes that they would go up still further. Those speculators are now sitting on the sidelines, afraid to get their fingers burned.

They see the hugh amounts of oil being stockpiled by the industrial West -- 1 million barrels a day is being squirreled away, according to the 20-nation International Energy Agency here. The traders and the official analysts assume that this must stop soon if only because storage capacity must be reaching limits.

"Nobody knows exactly how much storage capacity there is in the world," said an IEA analyst, "but there is an awful lot of slow steaming going on." This is the practice of keeping oil tankers loaded at sea or in port for longer than usual because storage tanks on land are full.

Iran has been a major factor in the stockpiling by the noncommunist industralized world of more than 110 days' worth of oil imports, and more than 150 days' worth in the United States. Governments and companies are obviously afraid that Iranian oil production, already down to 3.5 million barrels from almost 6 million barrels a day before the revolution, could be cut off altogether.

The Iranians, meanwhile, have made it much harder for the West to keep track of what is going on in the market. They are selling to more than 40 big and little clients, many of them inexperienced newcomers, who can be pitted against each other, instead of, as before, to a stable and united consortium of companies.

Iran had been demanding that companies with contracts pay spot prices for 15 percent of their Iranian supply and also pay $5 a barrel on every barrel currently delivered as an "advance payment" if the companies expected more oil in 1980. This drove the current Iranian price for some companies up to as much as $40 a barrel.

The formula the Irnaians seem to have failed to enforce this week was a 50-50 deal for half the oil at Iran's "official" price of $28.50 and half at a fictitious spot price of about $40 averaging out to about $35.

Other oil exporters that have been applying such "spot" prices to part of their production include Iraq and Libya.

The producer's spot prices are justified by the highest quotations those countries find in Platt's Oilgram, published daily by McGraw-Hill. Nobody is really sure what the Platt's prices really represent on any given day.

Western analysts concede that Platt's is faily representative, and its journalists insist that they speak frequently enough to a cross-section of several hundred companies in the market that it is hard for them to be fed phony prices.Many professional observers say privately, however, that Platt's often enough publishes offering prices at which there are not takers. "I've never been able to get clear in my own mind," said a close student of the industry, "whether Platt's reflects the market or makes the market."

Another major source of uncertainty that has driven people into the spot market has been the tendency of producing countries to take oil away from the major multinational oil companies and sell it themselves in state-to-state deals. Six million barrels a day are currently being sold state-to-state, with Japan's large state-guided trading companies as the biggest buyers.

The Saudi national company Petromin, for instance, disposes of 2 million barrels a day of the country's 9.5 million barrel production. Saudi Arabia has been producing 1 million barrels a day more than its self-imposed limit. When it goes back down to 8.5 million barrels a day, the four American oil companies that make up the Aramco operating group will be squeezed back by another million barrels a day.

IEA analysts calculate that the majors have lost access to about 3 million barrels a day this year, creating chaos in the established patterns of the world oil trade.