The United States and the Soviet Union may be moving toward settlement of a longstanding dispute that has delayed formal negotiations to constrain nuclear testing, U.S. officials said yesterday.

Signs of flexibility have emerged in private discussions here between Soviet Embassy officials and Arms Control and Disarmament Agency director Kenneth L. Adelman, officials said. Several officials said they hope for an informal agreement on the agenda of the talks in time for a mid-September meeting in Washington between Secretary of State George P. Shultz and Soviet Foreign Minister Eduard Shevardnaze.

The talks are being conducted against a backdrop of renewed cooperation between U.S. and Soviet scientists who want the two sides to restrict nuclear tests, and also amid fresh charges by the State Department that the Soviets have violated a 1963 treaty barring the emission of radioactive debris from nuclear tests.

Eight U.S. scientists participating in a program cosponsored by the Soviet Academy of Sciences and an environmental organization here, the Natural Resources Defense Council (NRDC), departed yesterday for a remote Soviet coal-mining area, where a large non-nuclear explosion is scheduled Sept. 2 to assist nuclear monitoring efforts.

The superpowers have long been at odds over the future agenda for nuclear testing negotiations, which the Reagan administration suspended in 1981. The Soviets have emphasized the need for a comprehensive testing ban, while the Reagan administration has been interested primarily in new measures to verify existing treaties that limit nuclear test explosions to 150 kilotons, equivalent to 12 times the force of the bomb that destroyed Hiroshima in World War II.

A senior U.S. official said the Soviets informally agreed in the private talks to take up the verification measures first, followed by discussion of "interim steps" leading toward a comprehensive test ban. U.S. officials, in turn, stated explicitly that the "interim steps" could be negotiated at the same set of talks once verification measures were agreed upon.

Several officials noted the two sides remain at odds over whether the agenda should specify that interim steps can only be taken if substantial reductions are first made in offensive nuclear forces. The Soviets have resisted such conditions.

The scientific group, which included experts from the University of Nevada-Reno and the Scripps Institution of Oceanography in California, plans to measure the shock wave from a Soviet non-nuclear explosion with a force of up to 10 tons of TNT, or one-tenth of a kiloton. The blast is to occur near the town of Karaganda, about 360 miles west of the principal site of Soviet nuclear tests.

Reps. Thomas J. Downey (D-N.Y.), Bob Carr (D-Mich.) and Jim Moody (D-Wis.) plan to witness the explosion, along with Anthony Battista, a staff member of the House Armed Services Committee. Downey has been a leading proponent of legislation to bar all but the smallest U.S. nuclear explosions, which the House passed this year.

The aim of the Soviet blast is to improve the accuracy of seismic measurements taken by U.S. equipment installed by NRDC near the test site. A similar blast will be conducted in Nevada early next year, according to NRDC scientist Thomas Cochran, who added that the seismic equipment could be eventually be used to verify compliance with new testing limitations.

The new U.S. allegation of a Soviet treaty violation concerned a nuclear test conducted Aug. 2 on the island of Novaya Zemlya in the Arctic Circle. According to a statement read by State Department spokesman Charles E. Redman, radioactive debris from the explosion was detected outside Soviet territory in violation of the 1963 Limited Test Ban Treaty.

The Swedish government, which recorded heightened but not hazardous radiation levels in several cities, has asked the Soviets to explain the incident.

Similar charges have been leveled by the United States before, based on data collected by a network of Air Force planes that fly near Soviet borders after nuclear explosions, seeking information on Soviet bomb components.