The Reagan administration plans to confront Soviet negotiators in Geneva next week with a charge that a sophisticated radar installation being built in Siberia violates the 1972 anti-ballistic missile treaty, administration sources said yesterday.

Officials said the alleged Soviet violation, heatedly contested by Moscow, is to be raised in talks to emphasize its severity and a U.S. determination that the Soviets must be more compliant with past accords before new agreements are possible.

State and Defense Department officials reportedly agree that Secretary of State George P. Shultz should raise the contentious issue next week with Soviet Foreign Minister Andrei A. Gromyko.

If the two are going to discuss defensive and offensive arms limitations, an official said, it is necessary to talk about what has happened to existing defensive limits.

In 1983, U.S. intelligence reported that a large phased-array radar was being constructed at Krasnoyarsk deep in central Siberia. When the issue was raised in diplomatic channels, the Soviets said that, when construction is completed in 1988 or 1989, it will be clear that the radar is intended to track space vehicles, as permitted under the 1972 ABM treaty.

The administration, though, has taken the increasingly strong view that the radar is in the wrong place for such a function and that it is related to antimissile defense. The ABM pact forbids erection of antimissile radars, except "along the periphery" of each nation.

Former senior U.S. officials McGeorge Bundy, George F. Kennan, Robert S. McNamara and Gerard C. Smith cited the Siberian radar construction in a Foreign Affairs magazine article in November as "one quite clear instance of large-scale construction that does not appear to be consistent with the ABM treaty."

This statement took on particular significance because the former officials otherwise were sharply critical of administration programs and policies, especially President Reagan's Strategic Defense Initiative.

A Soviet-affairs specialist outside the government said that, although the Soviets have not been responsive, they might be willing to make some accommodation on the radar "if it appears the United States is serious about arms-control issues in which they have an interest."

The Soviets, he noted, have been complaining in tit-for-tat fashion about alleged U.S. violations of arms agreements.

Officials said Shultz also may raise the issue of Soviet encryption of telemetry, the encoding of radio signals from missile tests in space. The unratified SALT II strategic arms control treaty, which both sides have pledged not to undercut, restricts such encoding to some degree. The administration argues that Soviet encoding goes much further than is allowed.

In a related development, Brookings Institution experts described U.S. willingness to restrict defensive weapons in space as essential to progress in arms-control negotiations and perhaps even to success in next week's Shultz-Gromyko session.

John D. Steinbruner, Brookings' director of foreign-policy studies, told reporters that "space must be dealt with or there will not be any progress anywhere" between the two nuclear superpowers.

Steinbruner said he believes that it is "overwhelmingly in the interest of both sides" to forge some restrictions on direct military applications in space but that published reports of administration attitudes make him "pessimistic" that the United States will agree.

Raymond L. Garthoff, a former U.S. career diplomat and a member of the SALT I negotiating team, said the "key question" in the Shultz-Gromyko talks will be "the relations between the offensive and defensive in strategic-arms issues and what the United States will do on the defensive side."

"There is a possibility of getting the Soviets to agree to some substantial limits and reductions on the offensive side if we do agree to some restraints on the defensive side," Garthoff said. He added that, if the United States is not prepared to negotiate seriously on space-based defense, the arms-control dialogue faces "a possible collapse" as early as next week.

Helmut Sonnenfeldt, State Department counselor under Henry A. Kissinger, described the Geneva meeting as "talks about talks" that at best could lead to additional contact between Shultz and Gromyko.

He said he places major importance on whether the Soviets are willing to discuss restrictions on their defensive programs, including the disputed Siberian radar.

It is "conceivable," Sonnenfeldt said, that Gromyko will come to Geneva with specific demands that the United States agree to restrict antisatellite or antimissile programs in space, which would make for "a pretty rough meeting," although not necessarily its collapse.