Iran's central bank has issued instructions to all domestic banks not to trade in dollars, which would include refusing to accept payment for oil in dollars, a bank spokesman said yesterday.

A United States government source said, however, "There is still a lot of confusion and a lot of inconsistency in practice over there. We have indications they are still accepting dollars for some purposes, including payment for oil. But they certainly are trying to convince people they are not."

In a separate development, Saudi Arabian oil minister Sheik Ahmed Zaki Yamani said the five-country strategy committee of the Organization of Petroleum Exporting Countries is discussing once again whether to set oil prices in terms of a basket of currencies instead of the dollar.

Neither report helped the dollar, which fell yesterday against the Japanese yen and stayed close to its historic low against the West German mark.

While both the Iranian action and the possible OPEC action involve a rejection of the dollar, they are actually quite different in nature. Most experts think neither step would seriously damage the dollar unless, in the OPEC case, the change also signaled a shift in the way in which countries with surplus oil money planned to invest it.

The difference in the two is that Iran apparently will still price its oil in dollars but only accept payment in some other currency. OPEC would price oil in terms of some basket, or group, of currencies, while still accepting dollars for actual payment.

The only effect of the Iranian change -- other than a psychological one on exchange markets -- is likely to be a one-time drop of about $100 million in the need for dollars to finance international transactions, analysts said. That is small change compared to the billions of dollars worth of such daily transactions worldwide.

Reports from Iran indicate some companies buying oil have been paying for it with German marks, or in some cases with a combination of marks and French francs.

At a two-day meeting of the OPEC strategy committee in Dhahran, Saudi Arabia, Yamani said the question of setting oil prices in terms of a basket of currencies was under discussion but that it was "not new to OPEC. OPEC used dollars at various times in the past. Before the dollar, it used gold, then dollars again, then a basket of currencies, then back to the dollar."

Yamani said such a switch might be part of the long-range strategy the committee will propose, or that OPEC might decide to switch before the plan is completed. Either way, the switch would not be related to the actual price of oil but more to the "protecting of OPEC member countries from currency fluctuations," he declared.

During the periods when OPEC was using some other method of pricing, it still accepted dollars for actual payment. The Saudis, among others, have told U.S. officials there is no alternative to payment in dollars even if OPEC wanted to use another payment method.Similarly, there are reports even the Libyans have the same view.

If that is the case, the damage to the dollar from a switch to a basket of currencies would be largely psychological.

The more important question, analysts said, is whether the countries that have oil surpluses choose to invest their money in dollar-denominated assets -- such as deposits at U.S. banks -- or in those denominated in some other currency, such as German marks. Choosing a currency other than the dollar for this purpose would put continuing pressure on the U.S. currency in foreign exchange markets.

Meanwhile, Indonesian energy minister Subroto said in Jakarta he expects OPEC to increase oil prices later this month to compensate oil exporters for the decline in the dollar and the higher cost of goods bought from industrial nations.

But, he said, the increase will most likely be "gradual and systematic" rather than a single massive jump as advocated by some of OPEC's more "radical" members.

OPEC ministers meet in Caracas, Venezuela, Dec. 16 to consider prices for 1980.

In Dhahran, Yamani, as he has in the past, put the blame on Western nations for rising oil prices. "OPEC is not in control of prices," he declared. "The reason for rising oil prices can be found with the Western countries. The main reason . . . is that some of these countries are willing to pay these prices for petrol."

Several OPEC states have breached the $23.50-per-barrel ceiling OPEC set in June. Oil industry officials have expressed concern OPEC might jump its average price to $30 a barrel at the Caracas meeting, a further 35 percent increase in the current average.

The oil minister of Qatar, Sheik Abdul-Asiz Bin Kalifa al-Thani, apparently reflecting the views of Saudi Arabia as well, said, "We are for small, gradual price increases.

But Ezzedin Mabrouk, oil minister of Libya, whose country has breached the ceiling, said the market could absorb higher prices.

A year ago OPEC ministers were also citing a falling dollar and inflation in industrial nations as a reason for raising oil prices. The more than 60 percent in oil prices so far in 1979, of course, is the single largest reason for the surge in inflation in the U.S. this year and is also an important factor in the drop of the dollar.