H. R. (Bob) Haldeman is sticking to his story that Richard Nixon never told him to destroy the White House tapes that led to Nixon's downfall.

Haldeman, the onetime White House chief of staff, is imprisoned in the federal Correctional Institution at Lompoc, 60 miles northeast of Santa Barbara.

He has refused all requests for interviews, apparently because he doesn't want to take the edge off sales publicity for his book, which The New York Times will publish in 1978. But Haldeman did grant a two-hour visit recently to an old acquaintance, Joe Scott, who publishes a well-informed California newsletter called The Political Animal.

In keeping with the ground rules of the visit, Scott did not quote Haldeman directly. But he did print comments Haldeman made to him in an earlier interview.

Scott said Haldeman told him then that Nixon came to him in April, 1973, and said, in part: "Maybe we should destroy all the tapes, excluding all those covering national security matters."

Haldeman told Scott he argued against that, contending that the tapes would give Nixon accurate knowledge of what he actually said "and a stronger basis to combat wild accusations."

"The President did not instruct me to destroy any tape, or to remove the taping system," Scott quoted Haldeman as saying in the 1976 interview. "And I did not do so."

Nixon said in his final television program with David Frost that he recalled telling Haldeman to destroy the tapes. Nixon indicated that he believed he would have finished his term in office if that had been done.

The Political Animal to be sent to subscribers this week says that Haldeman works seven days a week as a laboratory technician at the prison camp's sewage disposal plant. His shift is from 2:30 p.m. until 9:30 p.m. with a half-hour off for dinner.

Scott says the job gives Haldeman "considerate privacy since he toils alone and is usually able to complete his assigned tasks in ample time for reading and writing."

At 50, Haldeman is one of the oldest of the 420 inmates at Lompoc, which during the World War II was an Army base known as Camp Cook.

Scott gave this picture of Haldeman in prison:

"Haldeman, who now sports a broad Pancho Villa-style mustache, looked somewhat thinner than before his incarceration. He keeps in shape by walking about a mile and a half to and from work each day, running two miles daily around the track of a rundown athletic field and playing an occasional game of tennis in the morning at a rather uninviting court behind the prison dormitory where he is assigned to a bunk in a section with 32 other inmates."