As Energy Secretary James D. Watkins pointed to the colored charts he used in his fiscal 1992 budget briefing last week, he said he was most worried about the costs depicted by the purple streak and the red wedge.

The purple streak represented money to be laid out for construction of the Superconducting Super Collider, an $8.3 billion experimental particle accelerator the Energy Department is planning to build near Dallas. On Watkins's chart, the purple section showed the annual cost rising from $243 million in fiscal 1991 to a proposed $534 million in 1992 and still more in subsequent years.

The red wedge showed the cost of cleaning up the nuclear and toxic wastes and restoring the environment at the department's huge, 12-state nuclear weapons manufacturing complex. The costs of the cleanup vary according to which Energy Department activities are included, but by current calculations the price tag has risen from $2.3 billion in 1990 to $3.5 billion in the current year to a projected $4.2 billion next year. The costs will approach $5 billion a year by 1996.

Waste cleanup and construction of the collider are by far the fastest-growing components of the Energy Department's $18.6 billion budget request -- "a mountain building up," Watkins called them. Because they will continue to rise in future years while overall spending remains relatively flat because of the spending cap negotiated in last year's budget agreement, Watkins said, all the other programs, such as energy research, are likely to be shortchanged.

"I'm not going to discuss '93," he said. "I don't know how to handle '93 right now. If we have to trim science and research projects, the nation will be the loser."

But even at the fast-rising rate projected by Watkins, spending on waste management and environmental restoration at the weapons complex is lagging what critics say is needed -- and what the Energy Department proposed to spend when it released a five-year cleanup plan last summer.

By their calculations, the department is planning to spend as much as $1.1 billion less on environmental cleanup next year than projected in the five-year plan.

The proposed budget "shortchanges environmental restoration and waste management by more than $1.1 billion less than the . . . five-year plan indicated would be needed," said Sen. Timothy E. Wirth (D-Colo.), a leading congressional critic of the Energy Department's cleanup efforts.

"We want to make sure the money is appropriated that's called for in the cleanup plan and that's not what's happening," said Michelle Merola, executive director of Concerned Citizens for Nuclear Safety of Santa Fe, N.M., one group in a nationwide chain of organizations monitoring the weapons plant cleanup.

Energy Department officials, including Watkins and cleanup director Leo Duffy, have argued that they are spending money as fast as they can usefully do so. They say they are reluctant to commit money to unproven technologies and that a shortage of trained waste management and environmental waste personnel makes it difficult to accelerate the pace of the cleanup.

Critics of the Energy Department have been complaining that funds budgeted for environmental cleanup are actually being used to bring some of the troubled bomb factories back into production. Duffy has rejected that charge, saying that money spent on safe retrieval and disposal of plutonium, americium and other dangerous wastes is essential to cleaning up the plants as well as to operating them.

No one disputes the fact that the task of disposing of the nuclear and toxic wastes at the nuclear plants, cleaning up the soil and water around them and restoring the environment represents a huge fiscal time bomb.

The Energy Department has said the task will take 30 years and cost many tens of billions of dollars. Stephen I. Schwartz, legislative director of Greenpeace, said last week that "next to the bomb factories' cleanup, the savings and loan bailout will soon look like loose change."

The cleanup involves such tasks as disposing of radioactive and toxic liquids fermenting in huge, corroding tanks at the Hanford, Wash., reservation and removing plutonium from fuel storage pools; stopping the underground migration of radioactive substances into drinking water supplies near some of the plants; and removing toxic metals such as chromium and mercury from the soil at the Idaho National Engineering Laboratory.

In a report released yesterday, the congressional Office of Technology Assessment said bluntly that the Energy Department may not be the right agency to manage this huge task, partly because of its shortcomings and partly because the public does not trust it.

The department's "stated goal -- to clean up all weapons sites within 30 years -- is unfounded because it is not based on meaningful estimates of the work to be done or the level of cleanup to be accomplished at the end of that time," the report said.

It said the department lacks scientific evidence to support its contention that the factories present no imminent public health danger, adding that "the technical and institutional resources and processes to make and implement sound, publicly acceptable decisions" are not in place.

Watkins said the OTA report confirms what he has been saying all along: "This is a problem of enormous proportions and will require yet unavailable technology and trained personnel to resolve it."