The United States yesterday formally demanded compensation from the Soviet Union for the deaths of U.S. citizens killed when a Korean Air Lines passenger jet was shot down by Soviet fighters, but the Soviets refused to receive the demand.

The diplomatic standoff came as the Federal Aviation Administration released transcripts and recordings of routine radio conversation between the pilot of KAL Flight 007 and U.S. air traffic controllers as the plane flew from Anchorage toward Seoul.

The transcripts gave no indication that either the KAL pilot or the controllers in Anchorage knew that the Boeing 747 was straying off course and into Soviet airspace.

All 269 people on board the plane, including 61 Americans, are presumed dead.

The U.S. demand for compensation was presented yesterday morning by John Kelly, acting assistant secretary of state for European affairs, when he called in Oleg Sokolov, the Soviet Embassy's deputy chief of mission.

Kelly presented Sokolov with a note demanding an unspecified amount of compensation, but Sokolov refused to accept the note, according to State Department spokesman Alan Romberg. Kelly then "refused to accept Mr. Sokolov's rejection of the U.S. note," Romberg said. He added that the note would be "re-presented," but did not explain how.

A similar note presented by Kelly on behalf of the South Korean government also was rejected, Romberg said.

The United States said in its note that it considers the shooting-down of the Boeing 747 "a flagrant and unjustifiable breach of applicable principles of international law," a reading that makes the Soviet Union responsible for reparations, it said.

Romberg said the government action would not prevent individuals from filing claims on behalf of the victims. If and when the Soviet government provides compensation, Congress would determine how the money should be distributed.

Romberg said citizens of 13 countries were on the plane. The other 12 are South Korea, Japan, Taiwan, Hong Kong, the Philippines, Australia, Malaysia, Thailand, India, Canada, Britain and Sweden.

The transcripts of the conversation between the KAL pilot and air traffic controllers at both the Anchorage tower and the oceanic control center in Anchorage provide no clues to what happened. The transmissions seem entirely routine.

On several occasions, the controllers could not reach Flight 007 by radio, but necessary instructions and information were relayed by the crew of another KAL flight on the same track. Such gaps in radio coverage are common on the high-frequency radios used for transoceanic comnmunications, according to FAA experts.

Nonetheless, the problems add credence to the suggestion that the plane's radios might not have been working properly. The pilot who flew Flight 007 from New York to Anchorage reported problems with an on-board radio and compass, an airline maintenance official told the Anchorage Daily News.

The equipment was checked in Anchorage and determined to be working normally, the official said.

Air traffic control radar coverage extends only about 165 miles from Anchorage; U.S. air traffic control responsibility extends for more than 1,500 miles. Once a plane leaves radar range, a matter of a few minutes with jetliners, controllers have no way of confirming its position. Pilots use on-board navigation equipment to find their way and are required to report by radio when they reach certain geographic positions.

In related matters:

* Reagan administration officials said they are persuaded that the KAL pilot did not see warning shots the Soviets claim to have fired because the crew of Flight 007 did not radio any trouble reports to controllers. "I have to assume" that the earlier radio difficulties were considered in that assessment, a U.S. official said yesterday.

* White House spokesman Larry Speakes reiterated that it was "practically irrefutable" that the Soviets knew they were shooting down a civilian passenger plane, not a spy plane.

However, administration sources also confirmed a Newsweek report that, shortly before the KAL plane was shot down, a Soviet air defense officer notified a Soviet surface-to-air missile battery that a U.S. RC135--a reconnaissance plane--had penetrated Soviet airspace.

U.S. officials say that the RC135 already had returned to its base 1,000 miles away when the KAL plane was shot down.