Millions of Americans have opened up individual retirement accounts since the tax-exempt savings funds were first made available three years ago, but a report released yesterday said that if Congress made some changes in the program, millions more would take advantage of the accounts.

The Investment Company Institute, which helped spearhead the drive to open up the program to just about everyone who works, said yesterday that the program would be much more attractive if Congress expanded the spousal IRA, permitted additional nondeductible contributions and allowed limited access to the account without penalty.

Contributions to IRAs during the 1983 tax year added $10 billion to U.S. savings -- money that would have otherwise been spent, according to the ICI, an association of mutual funds. IRA assets climbed from $26 billion at the start of 1982, when nearly everyone with earned-income became eligible for the plans, to $132 billion at the end of last year. About 23 million households have opened the accounts since 1982.

Additional 1984 IRA contributions can be made up until April 15. The institute said that, if these trends continue, at least another $10 billion will be added again.

David Silver, the institute's president, said the survey "confirms the astounding success of the IRA program."

"In addition to providing millions of Americans with the opportunity to plan for retirement, this do-it-yourself program allows each IRA owner to choose from among the full range of savings/investment options available, ensuring an undistorted flow of stable, long-term capital to finance industry, commerce and agriculture," Silver added.

Despite the rapid growth of IRA assets, about three out of five households in groups most likely to open an IRA have not yet done so, the ICI report concluded. The groups most likely to open an IRA are heads of households under age 65 with household incomes of $15,000 or more.

A barrier to investment in an IRA is the limitation of a spousal account for non-working spouses, ICI said. If only one spouse works, the couple is limited to a combined $2,250 contribution per year.

Allowing nondeductible contributions would entice about 6 million more households to invest, ICI said. Currently, no more than $2,000 per year can be contributed to an IRA. Those contributions are tax-deductible, and all earnings accumulate tax-deferred until withdrawal.

Limited access is a further obstacle to opening an IRA, the association siad. If people were permitted to withdraw IRA funds without penalty for the purchase of a home, education expenses or financial emergencies, 19 million households would be more likely to open an account, based on the survey.