THE OPPOSITION that President Bani-Sadr is encountering to his effort to end the hostage affair provides unwelcome evidence that his writ is still under some challenge in Tehran. The militants, encouraged by fresh word from Ayatollah Khomeini, continue to demand the return of the former shah and his money. Whether this will actually block the release of the hostages will be known only as the negotiations unfold. The uncertainty is not reason to give up hope, but to proceed with due care.

From the outside, the negotiation is hard to penetrate. President Sani-Sadr has yielded on the initial Iranian demand to retrieve the shah before giving up the hostages. President Carter has yielded on his earlier insistence on retrieving the hostages before accepting a process to review Iran's grievances. A U.N. "commission of inquiry" has been formed. But it is not yet clear whether it is supposed to find facts or pronounce judgments, whether it will focus on the shah, the United States or the hostages, and what its specific connection is to freedom for the captives. Toying with the hostages at this point and in this connection, Iranians should understand, would be intolerable. The panel's only legitimate purpose in meeting the hostages at all is to verify their number and see to their medical needs.

To some Americans, the whole process of bargaining for the hostages is an unbearable humilation. But Jimmy Carter early on rejected that notion. To judge in particular by the relative civility of the campaign debate on the issue, most Americans agree with him that every feasible diplomatic opening must be explored. That means, however, that at the end Mr. Carter will be faced with an excruciating decision. It is possible to see what it will be about. The president has already begun to cope with Iran's interest in a pledge of no future intervention. The difficulty lies in Iran's demand for some statement of guilt or responsiblity for American policy in the past.

Frankly, we do not think the president needs much advice in this matter. His tactics may be open to second-guessing, but his choke point, we estimate, is about where it ought to be. What troubles us is that the Iranian demand is being made by people with an imperfect understanding of, not so much the American political scene, as the American spirit, the kind of people Americans are Mr. Carter, like his countrymen, desperately wants the hostages back. He will pay a price, a very disagreeable one. Quietly, we think, people are facing up to it. But there are limits to the national humiliation Mr. Carter can accept for the sake of the hostages. If Mr. Bani-Sadr wishes to end the crisis, he will respect those limits.