The Reagan administration pressed the Soviet Union yesterday to take constructive steps in the wake of Sunday's shooting death of a U.S. Army officer in East Germany, and announced that it is considering retaliatory steps if it receives no satisfaction from Moscow.

The administration also released new details about the killing of Maj. Arthur D. Nicholson Jr., saying that he was attempting to photograph Soviet military equipment in a garage-like storage shed when he was accosted by the Soviet sentry who shot him.

Officials maintained that Nicholson was permitted to make an effort to take such pictures under the longstanding ground rules of what they called a "cat-and-mouse" game played by U.S. and Soviet intelligence-gathering teams in Germany.

The United States has made the reconfirmation and improvement of these ground rules a major objective in the light of the shooting. They were reported to be the focus of U.S.-Soviet discussions late yesterday at the State Department.

"If [Soviet leader Mikhail] Gorbachev wanted to send a signal to the United States that the Soviet Union is ready for an improved relationship, this would be one area for a signal to be heard loud and clear," said Assistant Secretary of State Richard R. Burt in testimony about the killing before the House Foreign Affairs subcommittee on Europe and the Middle East.

Several hours later Burt met with Oleg M. Sokolov, the second-ranking official of the Soviet Embassy here, to discuss growing U.S. displeasure at Soviet handling of the case and the need for remedial measures. If the Soviet attitude is constructive, officials said, a meeting of top U.S. and Soviet commanders in Germany is likely to be convened to work on ground rules to prevent another such killing.

White House spokesman Larry Speakes said that the administration is "considering some steps involving U.S.-Soviet relations" in connection with the shooting, but that no decisions have been made. This statement evidently was intended to make it clear that failure to respond in a satisfactory manner will have consequences beyond the military arena in divided Germany.

As a first step, officials said, the Pentagon Tuesday broke off a West Coast tour by the senior Soviet military attache in Washington, Rear Adm. Ivan Sakul'kin, and summoned him to the Pentagon to receive a protest about the shooting.

One official said other senior Soviet military attaches here are being summoned to meet with their Pentagon counterparts for similar protests.

The retaliatory steps suggested in case the Soviet attitude is deemed unacceptable span a broad range, submitted for top-level consideration by a variety of U.S. agencies and bureaus, an official said.

He and other officials spoke of their hope that the incident will not seriously dampen Washington-Moscow relations at a time when a new Soviet leader representing a new generation gives promise of improvement.

It was clear, however, that U.S. officials increasingly see the handling of Nicholson's death as a test case of relations in the Gorbachev era, especially in view of Soviet public statements Monday and Tuesday that they said made the situation "more difficult" rather than contributing to a solution.

Burt made clear in an initial discussion of the case with Sokolov Monday morning that the United States, while strongly condemning the shooting, was looking for ways to manage the situation, according to U.S. sources. This was in keeping with an initially low-key U.S. response decided upon at senior levels of the government late Sunday and early Monday.

A Soviet Embassy statement to Washington journalists Monday and especially a dispatch from Moscow Tuesday by the Soviet news agency Tass pinned blame for the killing on Nicholson. Release of these accounts angered the administration, leading to U.S. public responses over the last three days.

Sokolov has been handling the case for the Soviet Embassy, U.S. officials said, because Ambassador Anatoliy Dobrynin has the flu.

Nicholson and his driver, Sgt. Jesse Schatz, operated in Soviet-dominated East Germany under a 1947 U.S.-Soviet agreement providing for an exchange of liaison personnel.

Burt told the House panel Nicholson had alighted from a U.S. military vehicle and was investigating a "shed" when shooting began. Administration officials did not dispute Tass' statement that Nicholson had "opened a window and taken pictures" of a storage area when discovered by the Soviet sentry.

Schatz remained in the vehicle and was observing Nicholson through a sunroof when the first shot was fired, Burt said. He added that the sergeant dropped to the floor and heard two more shots, followed by a cry from Nicholson that he had been hit.

When Schatz tried to leave the vehicle to administer first aid, he was waved back at gunpoint by the Soviet sentry, Burt said. It took 30 minutes for a Soviet medic to arrive and another 30 minutes for medical personnel to attend to Nicholson, by which time he was dead, Burt said.

Burt told the committee and other officials at a State Department background briefing that the Soviets had designated the area where Nicholson was killed as off-limits to U.S. liaison personnel but that this restriction was lifted in February.

Late yesterday, however, a senior Pentagon official said that this was in error and that a recheck of data showed that the area of the killing had not been off-limits to U.S. personnel at any time.

Liaison personnel on both sides attempt to photograph each other's military gear and activities and are free to do so outside of designated restricted zones, officials said.

As part of the "game," both sides seek to detain temporarily any liaison officer who succeeds in taking military-related pictures, U.S. officials said. U.S. and Soviet personnel have been detained frequently, but there is no history of or justification for use of lethal force in such situations, the officials said.

The Pentagon released its written instructions to U.S. military personnel in Germany to detain Soviet liaison personnel "observed photographing, sketching or observing U.S. troop installations or activities" even in nonrestricted zones. The instructions also say that "no force should be used or lives endangered" when making detentions and that military courtesy should be shown.