The Reagan administration, in a move that delights Virginia's Dalton administration and angers organized labor, is turning over all responsibility for occupational safety inspections to the state.

Virginia has been trying to assume responsibility for inspections since the early 1970s, but the federal government always found enough problems with the state's program to maintain some control. Now, the federal Occupational Safety and Health Administration is ending its inspection programs in Virginia, three other states and Puerto Rico because of budget cuts and a desire to fulfil administration promises to reduce regulation.

"We won't have the U.S. government looking down our collar. It's a victory for Virginia," said Clayton P. Deane, assistant commissioner of the state's Department of Labor and Industry.

The reaction of the state administration headed by Republican Gov. John N. Dalton was echoed by the building industry. "It's going to be good," said Robert Johnson, spokesman for the Northern Virginia Builders Association. "It should mean better safety for our people."

But Fairfax County Supervisor Audrey Moore (D-Annandale), who led the county's effort to abolish the much-criticized state occupational safety program after there were 14 construction-related fatalities in Northern Virginia during 1978 and 1979, said angrily: "Workers in Virginia have lost all protection on safety. There is absolutely no concern for construction workers in the Virginia office of occupational safety."

Organized labor was just as angry about the Reagan administration's decision, which was announced by OSHA chief Thorne G. Auchter. "It's cruel, it's unthinkable," said Paul A. Askew, president of the Central Labor Council of Tidewater in Norfolk. "Safety is gone. It's a sad day for the Virginia worker."

Like other labor leaders, Askew said he has no confidence in the Virginia occupational safety office to do an effective job. "The state of Virginia has neither the desire nor ability" to inspect working conditions at businesses and industries, he said. "People have to be dying before they'll come out and take a look."

Deane, Virginia's assistant commissioner of labor, vigorously defended the state's own safety program. "If you want to compare the number of inspectors we have with what the U.S. government has and the number of inspections we do and the U.S. does, you can see that we do far more than the federal government," he said.

In 1980, Deane said, the state occupational safety office, which has 38 field inspectors, made a total of 2,448 inspections. The 12 federal inspectors made a total of 782 inspections during the same period, according to federal OSHA records in Washington.

Deane, referring to criticisms of the state safety office by labor leaders and the Fairfax Board of Supervisors after the 14 fatalities in Northern Virginia in a two-year period, said: "We didn't have any more accidents in Northern Virginia, proportionately, then in the rest of the state."

Under federal occupational safety legislation, the states are supposed to assume responsibility for inspection after their programs are certified by the U.S. Department of Labor.

The OSHA's field coordinator, John B. Miles, said the Virginia program was good enough to exist without federal backup. "There have been problems in the past," he said, "but it's developing and improving. It's not one of the top programs, but it's a good one."

The federal shutdown will mean the elimination of five inspectors who cover Northern Virginia out of Falls Church.The state plans to add one inspector to its own five-person force in Northern Virginia, the state's busiest construction center.

The state's Deane said: "I don't think that causes me any concern at all. Our six people will be able to do an adequate job."