Sen. Arlen Specter (R-Pa.) warned yesterday that a Senate Appropriations Committee vote to delay construction of a D.C. prison in Southeast Washington while three alternative sites are reexamined could jeopardize the federally funded project.

Specter, the past chairman of the D.C. Appropriations subcommittee and a prime force behind the project, said the committee vote Thursday threatens to undo a "Herculean" four-year effort by Congress and District officials to cope with the problem of prison overcrowding.

"I'm very much concerned that if the prison gets off track, it will never get back on track," Specter said.

Meanwhile, Mayor Marion Barry also criticized the action, noting that Congress specifically mandated the site of the new prison in appropriations legislation approved last year.

"It is unfair to the neighborhoods and the wards of this city to continuously raise the specter of this issue year after year," Barry said. "The decision was made by the Congress and it should be held to its original decision."

Sen. Tom Harkin (D-Iowa), who succeeded Specter as subcommittee chairman, sought the delay after receiving numerous complaints from residents near the site of the proposed $50 million prison, adjacent to the D.C. Jail, and learning that a portion of the site contains historically significant artifacts.

"That site isn't totally foreclosed, but it looks like it isn't going to work," an aide to Harkin said yesterday.

City officials, who originally intended to break ground on the prison project last spring -- before an archeological study turned up findings that caused a delay -- now face a series of perplexing political and legal problems.

The city is under a series of court orders to reduce crowding in the jail and in Lorton Reformatory, and officials have promised to complete construction of a prison by 1990. If Congress rules out the current site, it would be impossible for the city to meet that deadline, city officials said, and the Barry administration once again would be faced with the politically sensitive task of finding an area of the city that would accept the prison.

In voting to halt work on the prison project, the Appropriations Committee instructed the Federal Bureau of Prisons and the General Accounting Office to report by Feb. 1 on whether the prison should be shifted to one of three alternative sites that the city previously considered, away from residential neighborhoods.

Two of those sites are along New York Avenue in Northeast Washington and the third is near South Capitol Street, on the south side of the Frederick Douglass Bridge, in Southeast.

Adding to the confusion created by this latest twist in the prison saga was a dispute over whether the federal government already had rejected the sites proposed Thursday by the Senate committee.

City Administrator Thomas M. Downs said Thursday that the three alternative sites had in fact been ruled out by various federal agencies in the past. For instance, the site near South Capitol Street, near Bolling Air Force Base, was found unacceptable to the Defense Department, Downs said yesterday. Other federal agencies that own the two other sites rejected their use, according to Downs, who also said one of the New York Avenue locations includes a brickyard that is a registered historic site.

However, Assistant U.S. Attorney John Oliver Birch, deputy chief of the office's civil division, disputed Downs' contentions, saying the Justice Department is the agency that has the authority to speak for the government on the issue and that the sites have not been rejected.

Downs said of Justice, "It was not theirs to reject or accept . . . . Other members of the Cabinet had control {of those sites}."

Meanwhile, D.C. Council member Harry Thomas (D-Ward 5) and council member Wilhelmina J. Rolark (D-Ward 8) consistently have opposed placement of a prison anywhere in their wards.

Engineering Science Inc., a consultant to the District, discovered signficant historic and prehistoric artifacts and deposits beneath a small portion of the current site that will become a parking lot.

Janice G. Artemel, cultural resources manager for the firm, said a deep sampling turned up more than 70 artifacts of an unknown prehistoric Indian tribe -- dating back 3,500 years or more. Closer to the surface, archeologists found remnants of 19th century buildings.

Artemel said her firm concluded in a report Sept. 10 that a portion of the prison site could qualify for inclusion in the National Park Service's Register of Historic Places, but that as an alternative the city could excavate the area and recover the artifacts within 45 days.