Taxpayers who visualize the Internal Revenue Service as a stone-hearted master sleuth backed by vast computers that ferret out any attempt to cheat would be surprised by a visit to an IRS "tax factory" such as its giant service center here.

While it is true that the IRS cannot function without computers, its operations depend on an antiquated, paper-based, manual card-filing model many decades old. This year, in its most fouled-up tax-filing season ever, it showed signs of cracking under a workload that mushroomed to 100 million returns involving historic complexity. And more problems loom.

This year's record backlogs of unprocessed returns, hundreds of millions of dollars of erroneous or delayed refunds, incorrect dunning notices, unjustified threats to seize property, unanswered inquiries and uncorrected errors have gradually subsided to nearly routine proportions, officials report. The costs in millions of dollars and lost credibility for the tax system are still being tallied.

But the season's bleeps and blunders were part of a larger, stickier dilemma that involves computers and is in some ways a classic conflict pitting the tortoises of bureaucracy and politics against the wild hare of technology.

Its symptoms are apparent at the Philadelphia Service Center, a plant the size of six football fields. One of 10 similar centers, it processes returns for the mid-Atlantic region, including the District of Columbia and Maryland, and was the site of some of the year's worst problems. By the end of July, 845,000 tax returns had not been processed, including 116,000 with refunds due. The center here is still recovering.

At the beginning of its processing "pipeline," workers sort and file incoming tax returns by hand. They sit at Victorian-looking contraptions called "Tingle tables." Named after their inventor, these are desks with tiers and tiers of wooden trays, designed to hold the various sizes and categories of paper tax returns.

Walls are lined with shelves of material filed in worn manila folders; more files are stacked on rolling carts. On one table is a huge pile of bank deposit slips, as much as $74 million for a single day's deposit during the height of the filing season.

Employes keypunch only certain information from each tax return into a computer system.

In the computer rooms, behind bulletproof glass, workmen with carts of color-coded wires install new units, or "black boxes." They are patching a new $103 million Sperry-Univac computer system whose nationwide implementation caused most of the problems of the 1985 filing season.

The system has proven inadequate to the workload, cost nearly twice what was expected and must be replaced soon.

Finally, after a complex procedure for ironing out errors, "good" taxpayer information is forwarded to the agency's National Computer Center at Martinsburg, W. Va. -- but not electronically. Magnetic tapes containing the data are moved back and forth by truck or plane, and the IRS tracks them using a paper-based manual filing system.

This occurs because, for more than a decade, lawmakers have cold-shouldered IRS pleas for a more sophisticated computer system, saying they fear that the agency might become Big Brother.

Watergate revelations that the Nixon administration targeted "enemies" for tax audits and general public suspicion of computers fueled concerns that a modernized operation would make such abuses easier.

In particular, lawmakers have prohibited the IRS from linking its computers in one great data base so they can "talk" to each other.

The General Accounting Office (GAO), Congress' investigative arm, is delving into the agency's problems, including acquisition and implementation of the Sperry-Univac system and related management deficiencies.

A House Appropriations subcommittee staff also is investigating the agency.

Congressional probers blame this year's chaos mostly on poor planning and misjudgment by top managers who chose and implemented the new computer system.

They also say a succession of top IRS managers, whose turnover rate is high and accountability diffuse, has failed to develop a coherent strategy for the agency's technological future.

IRS officials concede that the new system is inadequate and say it will be replaced by 1989. But they fix partial blame for the snafus and seeming lack of planning on the restraints imposed on them by "outside influences" -- Congress and the executive branch.

"That is the travesty in this situation. We are faced with dealing, unfortunately, with equipment that by the time we're able to replace it we are back behind the power curve," M. Eddie Heironimus, the IRS' top computer official, said in an interview.

A 25-year IRS veteran, Heironimus has been associate commissioner for data processing since early 1983, overseeing divisions dealing with computer services and processing of tax returns.

Some critics say the computer tangle and many other IRS problems, such as staff shortages and inadequate training, have worsened dramatically under President Reagan's budget-cutters, perhaps undermining the credibility of the entire tax system.

If the public loses confidence in the voluntary tax system, "all our work to reform the tax code will come to nothing, and there will be no hope of bringing the federal budget into balance," said J.J. (Jake) Pickle (D-Tex.), chairman of the House Ways and Means subcommittee that oversees the IRS.

The volume of income-tax filings keeps swelling -- from 6 million in 1930 to 97 million last year, from 100 million this year to an anticipated 200 million by 1990 -- and, with each new tax law, returns are more complex.

The IRS workload has increased by nearly half in the last decade, while the staff has grown less than 3 percent, to 86,500, on the assumption that automation would bridge the gap.

Despite its deficiencies and staggering workload, the IRS is by most accounts one of the most efficient and technologically advanced government bureaucracies. Once the computer equipment was running last spring, even with all its flaws, the agency set weekly records for nearly two months, reaching an all-time peak of 7.1 million returns processed in one week, IRS officials said.

The foul-ups are costly. Besides an extra $90 million to beef up the original $103 million worth of Sperry equipment, the IRS has paid $40.2 million in interest on late refunds this year, officials said. That is about $14 million more than it was paying at this point last year. The agency has also paid employes $22.3 million in overtime, about double the amount for this time last year.

Pressure on overworked employes contributed this year to rare but highly publicized cases in which workers destroyed or hid documents to meet production quotas, congressional investigators found. These occurred at service centers in Andover, Mass.; Austin, Tex.; and Fresno, Calif., the largest one. Other centers are in Atlanta; Brookhaven, N.Y.; Cincinnati; Covington, Ky.; Kansas City, Mo.; Memphis; and Ogden, Utah.

The greatest potential cost to the IRS cannot be calculated. It involves increased tax cheating by people who now believe "they can get away with it," as one official said.

Slightly fewer than 600,000 returns remained to be processed nationwide at the end of September, IRS officials said. An estimated 1 million refunds, some for returns already processed, had yet to be certified and mailed. Several thousand refund checks are believed still uncertified. This year's average refund was $836.78.

The computer equipment and the programs, the officials added, have "shaken down," and they do not expect the horrors of 1985 to be repeated.

But others express concern that the agency could be swamped again before it can resolve errors generated this season, or that it may encounter unexpected difficulties with the still-evolving computer operations.

In addition, the GAO, the National Treasury Employees Union and private tax experts say the agency is seriously understaffed and underfunded, that morale is low and turnover high, especially among those with valuable high-technology skills who can earn more money in the private sector.

For 1986, the administration proposed cutting $30.4 million from the IRS budget, authorizing 1,254 fewer staff positions and delaying plans to strengthen the tax examiners (audits) staff.

The Senate, for reasons that may have more to do with budget-process complexities than with the IRS, has voted to cut even more than the administration proposed, according to congressional aides. It did so in a drive to cut the deficit, they said, even though any investment in the IRS brings the government a profit -- as much as $16 for each dollar invested, depending on how it is used.

The House has approved $178 million more than the White House proposal, which is expected to yield $1.2 billion in increased revenues over three years. No date has been set for a House-Senate conference.

Trying to economize at the IRS is "slaughtering the goose that lays the golden egg," said Jerome Kurtz, its commissioner in the Carter administration, expressing the sentiments of many critics. "The IRS has been underfunded for a long time. It has become almost desperate in the last four years," he said.

The agency has "backlogs everywhere. Some of it replaces itself every day," an aide to Pickle said. "When we beat up on them" about a backlog in one area, the aide added, the IRS switches employes from other areas, "and then that lags."

Tax audits are being performed at the lowest rate ever -- 12 per 1,000 compared with 20 per 1,000 in 1980 -- while taxes owed but not collected have soared to more than $90 billion. The GAO told Congress this year that, if all taxes owed had been paid in 1981, the last year for which detailed estimates are available, "the government would have had a surplus of $2.5 billion instead of a deficit of $79 billion."

Last month, the National Treasury Employees Union revealed that the IRS has even eased its secret standards, or "tolerance levels," that determine how big a taxpayer debt or filing discrepancy it will pursue aggressively.

While the agency can reach into many private corners, it is more limited than many believe in its ability to cross-match information. For instance, it cannot check reported charitable deductions against contributions reported by tax-exempt groups. And its taxpayer-assistance program has deteriorated even though taxpayers need more help than ever in reconciling problems, according to congressional sources.

IRS Commissioner Roscoe L. Egger said he has ordered a senior internal task force to study the problems of the 1985 filing season, and he declared recently that auditing is "no longer our first priority. It's taxpayer service."

Many critics, however, feel that more money and skilled employes will not solve the agency's computer drift.

For elected officials, incentives to pump more money and clout to the pitiless federal tax collector are murky at best. As one Senate Finance Committee aide said, "Any time you step up enforcement, you increase public complaints."

Sen. John Heinz (R-Pa.) would support giving the IRS more money, an aide said, "if they know how to use it." He added, "We have taxpayers who have called us as many as 200 times" about an unresolved problem at the Philadelphia service center, which Heinz has called a "computerized chamber of horrors."

The advent of mini- and microcomputers has made the technology less threatening for many skeptics, so a more "computer-literate" Congress and public may be ready to accept increasingly sophisticated use of computers, with proper safeguards, IRS and congressional sources speculated.

Congress approved expanded use of confidential tax information last year to double-check the claims of possible food-stamp and other welfare-program cheaters, and the administration is pushing to broaden that approach to more federal programs. However, civil libertarians and some lawmakers continue to express concern about safeguards adequate to prevent misuse of the information.

"Since there have been so many foul-ups in the system, that makes us even more concerned about the use of computer data" and the possibility that someone's benefits might be stopped or reduced because of faulty information, said an aide to the governmental affairs oversight subcommittee chaired by Sen. William S. Cohen (R-Maine). There is also concern that disclosing tax data might undermine taxpayers' willingness to file accurate returns.

Still, some see the need for a beefed-up, streamlined IRS as one of essential fairness. Much "voluntary compliance" is built into the system through payroll deductions, Kurtz noted.

"So, it's a question of whether you want to finance the government on the backs of honest people and let the cheaters get away with more and more," he said.