The Reagan administration will keep up public pressure on the Soviet Union over alleged violations of arms-control agreements with a new report to be sent to Capitol Hill in December, according to a White House aide.

The report, mandated by Congress, will be "more significant" than the study of Soviet noncompliance since 1958 by a presidential advisory panel that was released Wednesday, the aide said.

An interagency committee, which is studying new intelligence data, he said, is focusing primarily on violation of the anti-ballistic missile treaty of 1972. Last January, President Reagan told Congress that the Soviets "almost certainly violated" that treaty with their deployment of a large phased-array radar in central Siberia.

The treaty restricts deployment of such radars to the borders of the country and requires that they face outward.

The Reagan administration considers as the most important compliance issues the continued Soviet encryption of radio-signaled data from its new missile tests, development of a second new intercontinental ballistic missile and the Siberian radar, administration sources said yesterday.

The United States will present its new data on the Soviet radar and its missile testing in a meeting with the Soviets of the Standing Consultative Committee in Geneva. The group, whose latest session began Oct. 2, was established by the original 1972 strategic arms limitation treaty as a forum for discussing alleged violations.

In earlier committee meetings and diplomatic exchanges between the two governments, the Soviets have maintained that the new radar would be limited to tracking objects in space, and not used to control anti-ballistic missiles.

The encryption issue has reached a deadlock, sources said yesterday. By intercepting radio signals from Soviet missile tests, U.S. analysts can determine whether the missiles are legal under the treaty. Recently, however, the Soviets have been encoding those transmissions.

The Soviets have asked the U.S. representative, retired Gen. Richard C. Ellis, former commander of the Strategic Air Command, to tell "what it is that you are missing," Air Force chief of staff Charles A. Gabriel told a Senate subcommittee earlier this year.

"If we tell them that," Gabriel went on, "they know what we can pick up and what we cannot." Therefore, Ellis says nothing in order to protect American electronic intelligence intercept capabilities, sources said.

Meanwhile, administration officials are studying two recent Soviet tests of a 15-year-old SS13 missile to see if Moscow was taking an unusual step to provide data to Washington to show it is not violating provisions of the unratified SALT II treaty, according to informed sources.

Under SALT II, the United States and the Soviet Union limited each other to testing and deploying only "one new type" of ICBM. But the small print of the agreement permits both sides to modify and thus modernize their existing missiles within certain parameters.

Reagan said last January that a new Soviet solid-fuel missile, the SSX25, "probably violated" the treaty because it represented a second "new type" of missile, since the Soviets had already said a weapon called the SSX24 would be its one "new type."

When Washington first raised questions about the SSX25, the Soviets said it was an allowed modification of the SS13. Ellis, the U.S. representative, persisted in saying its characteristics were different from the older missiles. The Soviets responded that the U.S. data on the SS13 must be wrong.

An administration official involved in arms-control matters said yesterday that he believed the SS13 tests may be related to the SSX25 issue. Pentagon analysts, however, found the SS13 tests did not remove concern about SSX25 being a profitable violation of the treaty, a spokesman said yesterday.