The Internal Revenue Service will tap the tax refunds of as many as 1.5 million Americans next year to help the federal government collect overdue debts for small-business, housing and education loans, the Office of Management and Budget announced yesterday.

Two weeks ago, it was announced that the IRS will help collect hundreds of millions of dollars in overdue loans made under the guaranteed student loan program. That action will affect as many as 1 million taxpayers, bringing to 2.5 million the number of Americans whose names could be turned over to the IRS to collect debts owed other agencies.

All of them, the OMB said in a statement, are "hard-core" debtors who could repay their loans. Before taking action, the IRS will wait 60 days after the taxpayer is notified to see if he pays. The money will be subtracted only from refunds the taxpayer would otherwise receive; the IRS will not attach anyone's wages, officials said.

Fred Wertheimer, president of Common Cause, the self-styled citizens' lobby, called the effort "real worrisome."

"The notion of moving the IRS out of its role of implementing and overseeing the tax laws to being the government's debt collector starts to move the IRS in a direction that could raise all kinds of questions dealing with rights of privacy, with mixed missions, with people's willingess to deal with the IRS," Wertheimer said. "They have run into their own problems recently in terms of their computer and carrying out the responsibilities they presently have."

OMB spokesman Steve Tupper pointed out that Congress authorized the IRS to try to collect bad debts. A two-year trial program was approved as part of the 1984 tax legislation. Tupper said OMB hopes to collect $253 million in fiscal 1986 and $344 million in fiscal 1987 through the effort.

Other federal loan programs could be added to the IRS list in the future, Tupper said. He added that the IRS program will not replace efforts to turn delinquent debts over to private collection agencies and screen applicants for federal loans to make sure they don't have outstanding debts.

These actions will "send the word to the public that we are not going to let people who have decided to be deadbeats get away with it," Tupper said. "I think what you're looking at is the start of a program rather than the culmination of it."

IRS spokesman Scott Waffle said his agency's role will be "purely technical."

The Housing and Urban Development Department, one of the agencies that will be turning names over to the IRS, announced that it will begin offsetting the salaries of some 500 HUD employes who are delinquent in paying back HUD home-improvement or mobile-home loans.

The five agencies that will turn names over to the IRS are:

*HUD, which has 25,000 debtors who have defaulted on $146 million in home-improvement or mobile-home loans.

*The Education Department, which is owed more than $1 billion by more than 1 million people who participated in the direct student loan or federally insured student loan program. The guaranteed student loan program, which involves another 1 million defaulters, is run through the states.

*The Veterans Administration, which is owed more than $200 million from 400,000 veterans who mistakenly received reimbursements for education and training.

*The Small Business Administration, which has 43,000 debtors who owe $124 million in low-interest disaster loans made as long as 10 years ago.

*The Agriculture Department, which has 12,000 outstanding rural, nonfarm loans worth about $51.8 million that the Justice Department is attempting to collect.

OMB said taxpayers who owe money will not be affected unless they receive a refund. On 1984 returns, the IRS said, the average refund was $833.