THE TELEPHONE callers mostly just wanted to reach a human being. People whose tax refunds were delayed, payments unrecorded or letters unanswered called me because they couldn't get through to the Internal Revenue Service.

"I've wanted to tell someone about my IRS problem for more than a year," said a woman whose payment was erroneously credited to a corporation and had a lien placed against a family trust. "I'm just a 69-year-old woman trying to do the right thing."

"They couldn't tell me how much I owed them," said a Wheaton man. "They kept sending me notices saying I hadn't paid and that they would attach my wages. They've hounded me for three years."

The horror stories are known now after widespread disclosure of IRS foulups, especially at the Philadelphia center where returns from the District and Maryland are sent. People harassed for payments that they had not only made, but authenticated. Social Security numbers recorded incorrectly. Liens placed on homes and property to collect taxes which had been paid years before.

What may not be as apparent is that the number of innocent victims nabbed by the IRS is almost undoubtedly going to keep growing as the federal government intensifies its tax-collection efforts.

New enforcement measures coming on line raise the possibilities of clerical errors that will sweep more legitimate taxpayers into the IRS net, and proposed cuts in the IRS budget are going to make it even harder for those taxpayers to prove they're not crooks.

Improved tax collections have been a high priority of the Reagan administration, but there is a point beyond which those efforts become counterproductive. IRS Commissioner Roscoe L. Egger Jr. himself admits that enforcement of the tax system has become increasingly fragile.

The consequences of the screw- ups, unless the agency takes extraordinary measures to rectify them, could be increasing tax evasion, even worse relations between the IRS and taxpayers and growing distrust of government in general.

Egger and other IRS officials say, quite rightly, that many of this year's problems have to do with new computer systems in all 10 regional centers. Because of inevitable bugs, the computers crash from time to time and IRS employes cannot call up taxpayers' files to answer questions. The computers are also largely responsible for the slow processing of returns. Nationwide, refunds will take an average of eight weeks this year, not including the time they are in the mail. In Philadelphia, the lag is 10 to 12 weeks.

But taxpayers don't care whether a computer or a person caused their problems if they can't establish communication or get their money when they expect it. People plan vacations and expenses around their refunds. "The IRAs have to be set up this week and I was going to use my refund for mine," said a Virginia caller who had sent his return to the center in Memphis in January. "They not only don't know where my refund is, they can't tell me anything. I'll have to borrow the money."

Every year for the last five years, returns have been sent in later and later in the filing season. That adds to the workload during the peak period and puzzles IRS officials. To me, it reflects the loss of faith in the IRS' promptness and shows how skeptical the public is becoming.

"If the IRS loses the idea that their people are public servants, they have problems," said Donald C. Alexander, who was IRS commissioner from 1973 to 1977. "If theyre going to take the position that all they have to do is circle the wagons, they're badly mistaken."

A number of new enforcement measures, while they may help find violators, could make other taxpayers even madder. For example, the government is now matching the forms of interest and dividends paid, with the same information filed by taxpayers. Similar reporting has begun for capital gains, state and local tax refunds, unemployment compensation, royalties and pensions. If either party gets a number wrong, whether it is a dollar amount or an identification number, the computer will flag it as a violation.

The IRS also is starting to withhold taxes on interest and dividend income for people who have not supplied their Social Security number, or who have underpaid in the past. Not surprisingly, some who have complied are seeing the money withheld by mistake anyway. Individual Retirement Accounts will give the service another chance to go after innocent people if the institutions holding those accounts mistakenly report the interest as taxable, as they have done on occasion.

People with accountants, time and money will be able to explain that a mistake was made. But more unsophisticated taxpayers will be likely to pay because fighting is too tough.

A 59-year-old Chevy Chase widow, for example, recently paid $350 to the IRS for the second time because she didn't want to hire a lawyer to continue her effort to prove she had already paid. The service was threatening to seize her property. (She did eventually get the money back, but only after months of letters and phone calls.)

Fewer and fewer IRS human beings may be around to answer those calls in the next several years. The administration is proposing to increase the number of IRS auditors by 7,500 beginning in fiscal 1987. But budget cuts are planned in other areas. The administration has proposed a decrease of $240 million for fiscal 1986 and a staff reduction of 3 percent in most departments, including the parts of the service that deal with taxpayer problems.

Egger said the new computer will free up more people to deal with complaints and that the agency is tracking system-wide difficulties to solve them before they happen. But some of his predecessors -- and the House Ways and Means oversight subcommittee that will hold hearings in two weeks on the adequacy of the IRS budget -- aren't sure that's enough.

"You need to have a balanced staff," said former comissioner Sheldon S. Cohen. "You cannot add just collection people, because every increase in that staff begets you more pieces of mail, more cases that go to court. When you rejigger and say, 'I'll just get more auditors,' that's silly."

Something is wrong when a taxpayer sends copies of his canceled check 10 times and still finds a notice on his door proclaiming that the IRS plans to sell his home for back taxes.

Something is wrong when the agency of government that touches more Americans than any single other entity just doesn't seem to be listening. That's bound to affect people's view of a tax system that still largely depends on voluntary compliance.