The Environmental Protection Agency is accumulating millions of dollars in its Superfund while, critics charge, it has done little to clean up thousands of hazardous waste dumps around the country.

But EPA officials insist the agency is moving swiftly to implement the trust fund, a program they say is technically complicated and requires substantial planning to avoid waste. They note that EPA has spent $19.8 million on emergency clean-up efforts at 54 sites, and has allocated $39.3 million for long-term remedial work at 40 sites.

Remedial work has been completed at only one of the 115 priority dumps that EPA identified last October. However, long term clean-up work has been started at three others.

Although the agency awarded a well-publicized $4 million grant last July to clean up the Love Canal, none of the money has gone out yet. Sen. Daniel Patrick Moynihan and Rep. John J. LaFalce, both New York Democrats, say remedial construction work there may be delayed until next spring because EPA has dragged its feet on completing a study that the agency says it must finish before work can begin.

Roughly 86 percent of the Superfund comes from quarterly taxes on chemical companies and 14 percent from the federal government. The trust fund is expected to collect $1.6 billion over the next five years, and the money can only be used to clean up hazardous waste sites and spills.

The fund had a surplus of $78 million at the end of fiscal 1981; it expects a surplus of $215 million at the end of fiscal 1982 and $352 million at the end of fiscal 1983. During fiscal 1983, it expects to collect $299 million in taxes on chemical manufacturers, $44 million from federal revenues and $23.5 million in interest, fines and recovery payments from private firms for cleanup efforts. EPA plans to spend $230 million, or about 40 percent, of the $582.2 million that will be available by the end of fiscal 1983.

"The only plausible explanation I can suggest for EPA's not spending the money is that the administration is looking for ways to offset the massive federal deficit," said Rep. James J. Florio (D-N.J.), House sponsor of the 1980 bill that set up the Superfund.

Hugh Kaufman, assistant to the director of EPA's hazardous waste control site, who has charged his agency with inaction, suggested another reason: "If by September, 1983, there's $900 million left, we can no longer collect Superfund taxes from industry."

William Hedeman, director of EPA's Superfund office, called Florio's suggestion "not only unreasonable, but absurd mathematics," noting the Superfund surplus is only a small percentage of the deficit. He said once projects proceeded from the planning to the construction stage, spending would boom.

EPA estimates the fund will pay to clean up 170 of the nation's 10,000 hazardous waste sites. Hedeman added that EPA expects to convince private firms to pay for work at many more dumps.

Letters are going out this month to negotiate private cleanup efforts at 100 of the 115 sites identified by EPA last fall as possible candidates for Superfund monies. Critics charge the negotiations could take months or even years, but Hedeman said if a delay seemed likely, the agency would proceed on its own, and possibly sue in court to recover triple damages.

Hedeman noted that most of the long-term, remedial projects are still in the planning stage. Kaufman said that is because "the administration has set up an elaborate red-tape system." Kaufman argued that the enormous amount of paperwork required by the current EPA is unnecessary, and in some cases enough information is already known for work to begin immediately.

Other critics place the blame for delays on the absence of an assistant administrator for solid waste and emergency response, who would have overall responsibility for the program. The Senate Environment and Public Works Committee is scheduled to hold hearings this afternoon on the controversial nomination of Rita M. Lavelle to fill that position. Lavelle previously worked as a public affairs official for Aerojet Liquid Rocket Co., whose parent company has been charged with dumping large amounts of toxic wastes in California ponds and swamps.