The John Weld Peck Federal Building, shown Tuesday, May 14, 2013, in Cincinnati, houses the main offices for the Internal Revenue Service in the city. (Al Behrman/AP)

The fog of scandal hangs over a boxy, modernist, 10-story building that looks like a monument to paperwork. Shrubs and chain smokers flank its front entrance here on Main Street, in the heart of downtown. Every day, 2,000 employees go to work at various federal agencies in this John F. Kennedy-era structure, whose chief tenant is the Internal Revenue Service — which is having just about the worst week an agency can have.

Up on the fourth floor — with its gray linoleum, low ceilings and fluorescent lights, file carts heaped with manila envelopes, its keypad-coded doors labeled 4-022 (the file room) and 4-034 (the supply room) — the determinations unit of the IRS’s ­exempt-organizations office is at work.

People in this Cincinnati unit have been accused of using “inappropriate” and “politically sensitive” criteria to scrutinize conservative groups that applied for tax-exempt status.

People in the unit, in a purple state’s red-leaning nook, have singled out applicants whose names include the words “tea party” and “patriot.”

People in Cincinnati have made people in Washington hopping mad.

“I am angry about it,” President Obama said at the White House on Wednesday, referring to the scandal that was also fueled by IRS offices in the District and California.

As could be expected, the folks in the determinations unit on Main Street have had trouble concentrating this week. Number crunchers, whose work is nonpolitical, don’t necessarily enjoy the spotlight, especially when the media and the public assume they’re engaged in partisan villainy.

“We’re not political,’’ said one determinations staffer in khakis as he left work late Tuesday afternoon. “We people on the local level are doing what we are supposed to do. . . . That’s why there are so many people here who are flustered. Everything comes from the top. We don’t have any authority to make those decisions without someone signing off on them. There has to be a directive.

The staff member, who spoke on the condition of anonymity for fear of losing his job, said that the determinations unit is competent and without bias, that it grouped together conservative applications “for consistency’s sake” — so one application did not sail through while a similar one was held up in review. This consistency is paramount in the review of all applications, according to Ronald Ran, an estate-tax lawyer who worked for 37 years in the IRS’s Cincinnati office.

“You’re not going to have a bunch of flaming liberals in the exempt-organizations department looking for conservative applications,” he said.

So what do you have on this fourth floor in Cincinnati?

An open, L-shaped layout of small, plain cubicles. (Office norms discourage the decoration of cubicle walls.)

“In movies where they put an IRS agent in a private office, and they’ve got all this stuff — I’m like, ‘In whose dream?’ ”

This is Bonnie Esrig. She retired from the IRS in January after 38 years. She spent her final year and a half in the determinations unit as an area manager who oversaw a roster of determination groups, each with 10 to 15 specialists who vetted applications for ­tax-exemption. Esrig says she was not in charge of any group that handled the conservative organizations in question, but she is surprised by the reports of “rogue employees” and by the public vitriol.

“I don’t think there was any malice or any kind of political” motive, she said. “I don’t believe that. . . . I just think there’s arrogant management that believed they were doing things correctly and didn’t really handle things as tactfully as they could have.”

This scandal developed, in part, because of a lack of bureaucratic “backbone,” said Marvin Friedlander, who worked in the ­exempt-organizations office for 40 years, trained many of the determinations staff members in Cincinnati and retired as head of the office’s technical unit in Washington, where IRS lawyers parse the murkier, more complex applications that are “elevated” from Cincinnati by lower-level specialists who aren’t necessarily certified public accountants or don’t necessarily hold accounting degrees.

These specialists must determine whether a group applying for 501(c)(4) status can prove that “social welfare” is its “primary activity,” meaning at least 51 percent of its operations. But regulations do not define how to measure this activity, according to Tuesday’s report from the Treasury inspector general for tax administration, and the technical unit did not provide guidance on some cases for many months, if at all.

“No one took charge,” Friedlander said. “No one saw the implications and the sensitivity. . . . In a bureaucracy, nobody wants to rock the boat. We don’t really want to tell them to stop the salami from getting made. So [the Washington lawyers] say, ‘We’ll keep working on this.’ ”

Friedlander, Ran and Esrig were not directly privy to the choices made in this episode, and details beyond the inspector general’s report are still emerging.

Nationwide, about 900 of the IRS’s nearly 100,000 employees deal with tax-exempt organizations. Cincinnati’s determinations unit handled about 61,000 applications last year. In recent years, office culture in Cincinnati has been defined by constant reorganization to offset a voluminous workload: Regulations in the Pension Protection Act of 2006 triggered a wave of reapplications, and between 2009 and 2012, the annual number of 501(c)(4) aspirants nearly doubled, to 3,357. Many had a political tinge that complicated the determinations process.

There has also been a reliance on transactional, ­numbers-focused management rather than ­consensus-seeking leadership, according to Esrig, who said she had little contact with IRS headquarters during her short tenure in determinations in Cincinnati.

“As an area manager, decisions were made and I would have to implement them,” she said. “But I didn’t often have an opportunity to provide input.

“You have to build support in the field and get people involved, and when you don’t do that, when people are just told what to do . . . ” she said, trailing off.

This brings us to IRS headquarters, built between 1928 and 1936 in the style of 18th-century French Renaissance architecture, next to the Justice Department on Constitution Avenue NW in the District. Its looming, columnar edifice looks like a monument to decision making.

Here, unlike in Cincinnati, IRS employees wear suits and ties, and can’t smoke near the entrance.

Here, unlike in Cincinnati, you can’t get into the building without an appointment.

Here is where IRS lawyers in the technical unit receive “elevated” paperwork from Cincinnati.

And here is where Richard M. Nixon attempted to use the agency to harass his “enemies list,” which created the perception that a totally centralized authority could not be trusted, said Milton Cerny, a lawyer who worked from 1959 to 1988 in various IRS jobs, including in exempt organizations. The Tax Reform Act of 1969 had begun to decentralize decision making to IRS field offices, including Cincinnati’s, and Nixon’s behavior encouraged this trend.

“There was concern that the national office might be taking too much control over audits and other activities in the field, [that] it should only be program direction, and not control,” said Cerny, who wrote a historical paper on the IRS’s functions between 1972 and 1992 titled “Tax-Exempt Organizations: It’s Been a Memorable 20 Years.”

Nixon’s name has been invoked by outraged pundits envisioning a high-level conspiracy, but the inspector general’s report implies that the problem here is a lack of control. The “inappropriate criteria,” the report said, were “not influenced by any individual or organization outside the IRS.”

In a way, Washington wasn’t meddling enough.

And this allowed middle managers in Cincinnati to make some bad calls.

IRS officials testified Friday in Washington before a House committee, the first of several expected congressional hearings on the issue. Higher IRS leadership is teetering: The acting commissioner resigned, and the commissioner of the agency’s tax-exempt and government entities division is retiring. And the lower level? The folks headed to work on Main Street in Cincinnati?

The media and popular culture can “make IRS agents out to be horrendous people,” said Esrig, the career IRS staff member who recently retired, “but they’re federal employees who work really hard for the most part. . . . They believe what they’re doing makes a contribution. . . . They’re not out to get you.”

Zak reported from Washington.