A three-year-old Federal Energy Administration program to force power plants to convert from oil and natural gas to coal failed because of "ineffective management" and "open warfare" in the agency, a confidential report charges.

The 21-page staff memorandum is the result of a six-week investigation by the House Commerce Subcommittee on Energy and Power which is reviewing President Carter's plan to step up the coal conversion program.

It offers a rare insight into the problems of a bureaucracy and the complexities of reshaping the nation's energy business. James G. Phillips, a subcommittee investigator, analyzed 400 documents and interviewed 23 federal officials for the report a copy of which was obtained by The Washington Post.

FEA's new administrator, John F. O'Leary, has acknowledged the problems of the program, designed to reduce the nation's dependence on Arab oil. "After three years, we haven't converted a drop of oil or a molecule of gas," he told the House committee last month.

"There hasn't been the constant dedicated management I would like to have seen," O'Leary said pledging to strengthen the program.

The subcommittee investigation revealed a history of angry "turf battles" between two FEA offices: the Office of Coal Utilization and the general counsel's office. Lawyers and non-lawyers fought bitterly over how much data the agency had to compile to prove that individual power plants should convert. Top administrators, viewing the program as a low priority, never stepped in to resolve the disputes.

Thus, the memorandum charged, so much time was spent trying to head off possible lawsuits, that almost no final conversion orders were issued.

At a hearing last month O'Leary acknowledged that the agency had lost files constituting "an essential element of the legal side of the case" for converting plants selected two years ago, but for which final orders were never issued.

"I think there is a black hole some where in FEA," he said. "People complain that documents come into my office and disappear. Every afternoon at four we regularly go around and look for some things.

"Over time the place has so much paper flowing around it, that it is literally awash in a sea of paper and crucial documents are mislaid and stolen or get lost. That has been a factor in the effectiveness of the agency."

According to the subcommittee report, "internal wrangling" over conversion stemmed from the law's vague requirements that conversions be economically practical, environmentally acceptable and that coal be available.

The general counsel rejected initial orders to 32 power plants in 1975, demanding that the Coal Utilization Office locate coal with specific sulfur, moisture, ash, volatility, ash softening temperature and heat content characteristics to meet the requirements of each power plant.

During an 18-month impasse over this issue, officials also argued over environmental strategy. The Coal Utilization Office deemed it sufficient to produce some overall environmental impact statement. The general counsel wanted extensive analyses of each power plant site.

Because this is such a time-consuming process, only five environmental assessments have been completed so far. Air and water quality, solid waste, wildlife and a dozen other issues must be examined in detail for each plant.

The lawyers said the agency must show conversion is economically reasonable on a plant-by-plant basis. The non-lawyers contended it was sufficient to demonstrate a utility's overall financial ability to absorb the cost.

The report also cited understaffing, low employee morale, frequent management turnover and "open warfare" between the director and deputy director of the Coal Utilization Office.

Carter's conversion program would shift the burden of locating coal from FEA to industry, but the agency would still have to prove the economic feasibility of conversion.

However Congress chooses to tighten the legislation. O'Leary has already begun to review management of the $12.8 million program. As a first step, he has asked the Justice Department to arbitrate disputes over legal issues.

Robert I. Hanfling, a deputy assistant administrator who now oversees the conversion program, said most of the problems described by the subcommittee have been resolved in the last few months and a new director has been hired."I wouldn't say there are no problems - there is a tremendous amount of work," he said. "But there are no more confrontations" between offices.