The Carter administration's new willingness to make gestures toward Iran that it publicly spurned three months ago, and the increased domestic strength of Iranian President Abol Hassan Bani-Sadr, appear to be the prime forces now moving to end the hostage deadlock.

These two elements, intertwined since the beginning of the crisis, seem only now to be coming into an uneasy balance that would permit a solution.

Bani-Sadr and at least two other actors in the complex diplomatic and political ballet that has centered around the estimated 50 American hostages in Tehran agree with the charge by Sen. Edward M. Kennedy (D-Mass.) that Carter's earlier refusal to consider the kind of inquiry he now accepts has helped prolong the crisis.

The other two are Rep. George V. Hansen (R-Idaho), the controversial self-appointed negotiator who sought to work out a deal for the hostages with Bani-Sadr in November, and House Banking Committee Chairman Henry Reuss (D-Wis.). Reuss said in a telephone interview yesterday that he had been prepared to hold hearings on the deposed shah's alleged misdeeds if it would win the hostages' release.

"Indeed this seems to be the thing the Carter administration is pointing toward now," Reuss said. "I applaud the president, wishing somewhat wistfully he had done it three months ago."

The Iranians and Hansen say they are firmly convinced that the White House's failure in November to outline hopes for better relations with Iran and hold open the door for an inquiry -- statements Carter did make this week -- robbed the then-isolated Bani-Sadr of the U.S. signals he needed to strengthen his hand.

Opposed to the hostage operation from the start, Bani-Sadr was acting foreign minister but was alone. Hansen, whose mission was dennounced not only by the White House but also by most of his congressional colleagues, felt he had convinced the Iranian politician that it was possible to hold an honest congressional investigation at which the militants' grievances could be aired.

Senior White House officials declined yesterday to comment on any phase of the hostage negotiations, citing fear of upsetting the current effort for a solution.

But the administration's senior policymakers were known to feel at the time that the earlier effort was aborted when Bani-Sadr was ousted by forces closer to Ayatollah Ruhollah Khomeini just as he prepared to come to the United Nations to discuss the crisis.

There were unconfirmed reports that the administration actually had been willing to communicate with Bani-Sadr, although not willing to make that position public, on the setting up of a U.N. commission as a step to the hostages' release.

At that time, U.S. officials were adamant in their public statements that the administration would not even discuss a commission of inquiry unless the hostages first were released.

Until Bani-Sadr's overwhelming presidential victory against hard-line opponents last month, those public statements apparently were what shaped Tehran's view of the U.S. position. Iranian officials say they responded to what they saw as White House pressure to scuttle congressional or international hearings on Shah Mohammad Reza Pahlavi.

Notwithstanding its unconventional aspects, the Hansen initiative for holding hearings involved two distinct advantages which finally became apparent to the Iranians despite ther initial suspicions of anything American.

Analysts then credited Hansen with a politician's understanding of Bani-Sadr's desperate need for a first U.S. gesture that would strengthen his hand and allow Iran to reciprocate.

Hansen also played on the U.S. separation of the legislative and executive branches to convince Bani-Sadr and the Revolutionary Council that if one congressional investigation could help unseat president Nixon, another could be counted on to offer an honest airing of their grievances.

Not enough is yet in the public domain to justify definitive conclusions about the administration's opposition to Hansen's proposal for congressional hearings or similar suggestions from U.N. Secretary General Kurt Waldheim after his early January visit to Tehran.

But analysts noted that both in Iran and the United States, the failure of Hansen's initiative led to more bellicose stands which only the Soviet invasion of Afghanistan eclipsed

President Carter dispatched the biggest naval task force to the Arabian Sea since World War II and his popularity began to outdistance that of the previous front runner, Kennedy.

In Tehran, Bani-Sadr lost his Foreign Ministry portfolio to arch-rival Sadegh Ghotbzadeh. The radical students holding the hostages increased their political leverage and began picking off pro-Western, middle-of-the-road Iranian politicians.

Chances for a settlement improved meaningfully only after Bani-Sadr won the presidency Jan. 25 and could speak from strength to begin whittling away the radical students' power base.

In November, Hansen's goal in addition to freeing the hostages was to have the banking committee, on which he served, investigate the role of David Rockefeller, chairman of the Chase Manhattan Bank and a favorite Bani-Sadr target.

Hansen took his case to Bani-Sadr, the Revolutionary Council and the embassy students and succeeded in winning their confidence. After initial opposition, the students allowed Hansen to visit about 20 hostages at the embassy, much to the administration's anger.

By telephone yesterday, Reuss recalled a Nov. 26 call from Hansen relating his progress. Thereafter, Reuss issued a statement saying he was willing to have his committee agree in principle to an investigation of the Shah's financial dealings, but that its actual work should begin only after the hostages were released.

Bani-Sadr welcomed the Reuss statement and said it could be the basis for the hostages' release. But the White House reacted immediately, strongly opposing Reuss' idea.

By telephone from Pocatello, Idaho, Hansen recalled yesterday that his understanding with Reuss was less restrictive. "There was no qualification that the hostages be freed first," he recalled, suggesting "it was more carte blanche."

Left unsaid was the hint that Reuss may have tightened up his conditions for opening the hearings after the White House heard of Hansen's initiative. f