The U.S. Patent and Trademark Office’s telework policy has made it one of the most popular places to work in government. It’s also made working from home a right instead of a privilege, leaving managers with virtually no authority to discipline employees who slack off.

That’s the conclusion of an independent review by the National Academy of Public Administration, which found that the 9,939 employees who work from home, most of them patent examiners, need more oversight. The eight-month review released Friday was carried out in response to a 2014 Washington Post report on a  cover-up by senior leaders of time and attendance fraud among patent examiners.

Supervisors told reviewers that the biggest obstacle to more oversight is the clout of the powerful employee unions, which have negotiated contracts with management that “make telework appear as a right and may present a significant management challenge,” the report said.

Management has worked with its unions in recent years “to get employee buy-in and foster a collaborative relationship,” the study found. And that has improved morale. But the academy also says that telework is not treated “as a privilege that is made available at the discretion of management.”

Dozens of supervisors interviewed said they have trouble knowing when an examiner who works for them from home is actually working, since the employee has to provide a schedule only of the hours he or she plans to work. The supervisors “believe there is a lack of consequences for poor performance and conduct violations, such as time and attendance abuse,” the review found.

It also found that the patent office, like other federal agencies that are increasing telework, needs a better way to measure the quality of work done, not just the quantity, given that the agency is heading toward a “virtual permanent workforce.”

In a statement, the patent office said the report confirms the “efficiency and effectiveness of the agency’s telework programs” and that the programs “substantially, quantifiably benefit the agency and the American innovators it serves.”

Managers have to acquaint themselves with the details of 29 different telework agreements governing employees, and that’s a challenge. The academy recommends that supervisors require “regular face-to-face meetings to engage teleworking employees and more effectively oversee their productivity and work quality,” even if the communication is through video conferencing.

Almost three-quarters of the supervisors interviewed said they would bring poor performers back to headquarters for additional training or coaching. But union agreements do not allow them to do that or to engage in the day-to day collaboration the report advocates.

The concessions that management has made to its unions “may have had the unintended effect of limiting [supervisors’]  ability to manage their workforce,” the report says. Some managers “expressly stated that they are unable to correct problems with employees because of union influences on the day-to-day operations of the Patent Organization.”

Read more: