The Department of Energy has told congressional investigators that it did not keep drafts of documents involving the selection of the nation's first high-level nuclear waste dump, and that its final recommendation to President Reagan contained a billion-dollar "typographical error."

The documents were requested last month by Rep. Edward J. Markey (D-Mass.), chairman of a House subcommittee investigating the selection process, who told Energy Secretary John S. Herrington in a letter yesterday that the missing documents suggest that DOE is either "engaging in a cover-up" or guilty of "incompetence."

Markey was seeking documents relating to DOE's decision two months ago to narrow the search for a high-level nuclear waste dump to three western sites: Yucca Mountain, Nev.; Deaf Smith County, Tex.; and Hanford, Wash. Questions have been raised about the environmental suitability of those sites, especially the government's nuclear reservation at Hanford, and whether political pressure played a part in the selection.

In its written response to Markey, the department said that "a number of working drafts were developed" before a final recommendation on the three sites was forwarded to Reagan in May, but that only the final version is available.

"When a new draft was produced, the previous draft was no longer operative or of any use, and was not retained," DOE said.

In general, government agencies are not required to retain copies of such internal documents unless a decision has been challenged in court. Legal challenges were expected in the case of the high-level waste dump, and all three states selected as finalists filed suit the day DOE's recommendation was approved.

Ben C. Rusche, head of DOE's Office of Civilian Radioactive Waste Management, said it is common practice in the office to dispose of outdated working drafts. "They serve no value," he said, adding that saving them is not "practical."

In a letter to Herrington, Markey and Rep. Al Swift (D-Wash.) said they found it "hard to believe that there is not one memorandum in DOE files . . . which relates to the secretary's decision to recommend the three final sites."

"We are left with only two possible conclusions: either the department is engaging in a cover-up and obstructing this subcommittee's investigation or incompetence has become the hallmark of this program," they wrote.

According to Rusche, there are no such documents because "I sent no formal documents to the secretary on this matter . . . . We don't work in such a manner that every communication with him is a formal document." Rusche said all communications between his office and Herrington's were oral.

"We talked about it from time to time," Rusche said. "There was no subterfuge involved. That's not the way we do business."

Markey and Swift also accused DOE of a "cavalier attitude" on the cost of the nuclear waste dump, noting that the department acknowledged that its recommendation to Reagan underestimated by more than $1 billion the cost of locating the dump at Hanford.

DOE initially estimated that it would cost $4.38 billion more to put the dump at Hanford, the most expensive site, than at Yucca Mountain, the least expensive. In its response to Markey's questions, DOE said the figure was a "typographical error" and should be $5.45 billion.

Rusche said he did not know the reason for the error in the original document, but said it may have been an error in arithmetic.

DOE also deflected Markey's request for a list of meetings and telephone calls between DOE officials and elected officials on the waste dump program, saying the department "does not create and maintain records of all such communications."

DOE did supply a list of meetings and calls recorded by Rusche's office, however. From Jan. 15 to May 29, when the department announced the three finalists for the first dump site and halted activity on selecting a second site, there were 50 such sessions, most of them involving members of Congress from states with potential waste sites.