Rep. Les Aspin (D-Wis.), reflecting a view held by others on the congressional Iran-contra panels, yesterday expressed puzzlement that only a small portion of the funds diverted from the Iran arms sales went for the benefit of the Nicaraguan contras at a time when administration officials worried about whether the rebels would survive.

Aspin told former national security adviser John M. Poindexter, who has testified that he personally approved the controversial fund diversion for the contras, that he found it "rather peculiar" that the rear admiral never verified how much the rebels actually received.

According to a preliminary analysis by the committees' staffs, about $3.5 million in funds diverted from the Iran arms sales was spent on behalf of the contras. But an additional $8 million from the sales was channeled into Swiss bank accounts controlled by private middlemen working with fired National Security Council aide Lt. Col. Oliver L. North.

"You look through these numbers and you come flat out to the conclusion that the idea that somehow the arms-for-hostages sale . . . produced enough money to keep the contras going during the 'dark days' didn't do it at all," said Aspin, who is also chairman of the House Armed Services Committee.

Aspin is the most recent member of the select committees to raise the idea that North's network of private middlemen may have stood to make excessive profits through the control of secret Swiss bank accounts used to receive funds from the arms sales.

Committee members have focused their criticism on retired Air Force major general Richard V. Secord, North's chief assistant in the Iran and contra operations, and Secord's business partner, Albert A. Hakim.

Secord has strongly denied that he and his associates were "profiteers" and has repeatedly said he renounced any profits owed to him for his work.

Sen. Paul S. Trible Jr. (R-Va.), an outspoken critic of the Secord and Hakim operation, charged again yesterday that by going outside normal channels, the administration set up a system without adequate financial controls. Trible, a contra supporter, said the evidence shows that "the contras were not well served."

"They were sent shoddy equipment at grossly inflated prices," Trible said. "And I think that underscores the folly of doing business in this fashion."

Poindexter said he did not confirm how much the contras were getting because he relied on the "trust and confidence" he had in North and Secord, adding he has not yet seen any evidence to question what they did.

Poindexter said he did not learn about the $8 million remaining in the Swiss bank accounts until it was disclosed by the committees.

North has said that he and the late Director of Intelligence William J. Casey discussed using some of the diverted funds to pay for future covert operations run outside the Central Intelligence Agency.

Aspin raised the possibility that the contras may have received funds from another source the committees have not yet uncovered. However, other sources familiar with contra financial records said there is no available evidence to support that view.

"If they didn't give over the $8 million during the" two-year cutoff of most U.S. military aid, Aspin said, "I don't know how you'd expect to think that they're going to hand it over during the fat days after" Congress resumed military aid to the contras last year.