The Senate Banking Committee decided yesterday to recall President Carter's nominee to head the Federal Reserve Board to interrogate him further about a questionable overseas payment his firm made in 1973.

The senators want to ask G. William Miller about a $2.9 million fee the Bell Helicopter division of Textron Corp. paid to an Iranian company controlled by the head of the Iranian air force . Bell sold 489 helicopters worth $500 millin to Iran in 1973.

Miller is chairman of the Providence-based conglomerate and was president of the company. with responsibility for Bell, when the payment was made to Air Taxi Co. of Iran in June 1973.

The Securities and Exchange Commission also is investigating the payment, a process which may take four to six months.

Although the banking committee's investigation of the payment has delayed Miller's confirmation by at least a month, Senate sources say that he will be confirmed easily both by the Senate Banking Comm, ittee and the Senate as a whole.

Nonetheless, it is the toughest sledding nominee to head the nation's central bank has ever had. It reflects the new-found vigilance in the Senate in the aftermath of Nert Lance's resignation as director of the Office of Management and Budget in a cloud of controversy over his banking practices while head of two Georgia banks.

A lengthy investigation by the staff of the Senate Banking Committee failed to turn up any evidence that Miller knew the company was making a payment to a firm secretly controlled by the late Mohammed Khatami, who was commander-in-chief of the Iranian Air FOrce when the half-billion-dollar helicopter deal was signed.

The Iranian company, Air Taxi, had been Bell Helicopter's sales representative in Iran. Miller told the Senate Banking Committee at confirmation hearings last month that the payment was made to cancel a sales representative agreement with Air Taxi, in part to compensate the company for work it had done for Bell in the past and in part to avoid having to pay future commissions to Air Taxi.

Although helicopter market in Iran had been minusucule until the early 1970s, Textron's helicopter division has made several billions of dollars of sales since then.

Miller said that neither he nor textron knew that Khatami, who died in hang glider accident in 1975, secretly controlled Air Taxi-which had been Bell's sales representative in Iran from 1959 until 1963 and from 1968 until 1973-when the agreement for military sales was cancelled. The helicopter sales were made under the U.S. government's foreign military sales program.

At issue is whether the payment to Air Taxi was in effect a bribe to General Khatami, who was close to the shah of Iran and who, according to the staff investigation, recommended Bell helicopters to the Shah twice.

Bell officials told committee investigators under oath that they did not know that Khatami was one of three individuals who controlled Air Taxi through nominees, although one said he might have heard the Khatmai still was affiliated with Air Taxi.

Khatami was one of three owners of Air Taxi until 1965, when the company ostensibly was reorganized because of conflict-of-interest rules issued by the shah.

James F. Atkins, president of Bell Helicopter-Textron, told Senate investigators he "probably" made the decision to pay $2.9 million on his own authority, although he kept Miller informed. As president of Textron, Miller at the time was de facto head of the aerospace group.

Next week the Senate Banking Committee will interrogate several Bell and Textron officials and is trying to get officials of Air Taxi to come to Wahsington to testify Miller is expected to testify again on Tuesday.

After yesterday's hearing, at which the staff report was summarized, chairman William Proxmire (D.Wis.) said he expected Miller would be approved easily, although Proxmire was inclined to vote against him because of his lack of experience in monetary and fiscal matters. "I may be a minority of one on the committee. I may be a minority of one in the Senate," Proxmire said. He noted that he was the only senator to vote against Lance.

The delay-Carter nominated Miller in late December-has been a slight embarrassment to the administration, but Proxmire said he has not received any calls from the White House on the Miller nomination since December.

Miller will succeed Arthur F. Burns, whose term as chairman was due to expire Jan. 31. Burns agreed to stay on as chairman until Miller takes over, but said he wants to leave the board all together by March 31 at the latest. Burns, who is 73, does not have to leave the seven-member board until 1984