President Reagan called yesterday for a "calm, controlled but absolutely firm" response to the Soviet Union for shooting down a South Korean passenger airliner, and high administration officials said this response probably would be limited to an international effort to condemn the attack and take measures to make it safer to fly near the Soviet Union.

Reagan, who earlier had decided against canceling the recently signed five-year agreement to sell American grain to the Soviet Union, appeared yesterday in the White House Rose Garden also to rule out any suspension of nuclear arms control negotiations with the Soviets, saying that "peace is that all-important that we should continue these talks."

One administration official said that Reagan, though outraged, saw no advantage in "retaliation for retaliation's sake" and did not need to convince anyone that he was distrustful of the Soviets. Another official said it was "highly likely" that the administration's primary response, if Congress agrees, would be to seek resolutions in international bodies "designed to combat this threat to international aviation safety."

Administration sources said the principal U.S. action likely is to be support for an emergency meeting of the International Civil Aviation Organization, a United Nations agency based in Montreal, that would consider improved warning measures designed to prevent planes from straying over the Soviet Union.

Korean Air Lines Flight 007, with 269 persons aboard, was 310 miles off its scheduled course when it was shot down, according to official Soviet estimates which the U.S. government has not challenged.

Among the proposals that have been discussed is installation of a radio beacon on Hokkaido, the northernmost island of Japan, that would provide a ground-based navigation signal to help pilots verify their location and the providing of military radar data to the civilian air traffic control system.

Other U.S. retaliatory actions are expected to be limited. A senior State Department official said that Secretary of State George P. Shultz will go through with his scheduled talk in Madrid this week with Soviet Foreign Minister Andrei A. Gromyko. But the official said Shultz will shorten the meeting and use it primarily to express a "feeling of revulsion" over the incident and Soviet unwillingness to accept responsibility for it.

Administration officials acknowledged that what White House spokesman Larry Speakes called "a measured response to the Soviet action" may fail to satisfy some of Reagan's most conservative political supporters. They said, however, that what the president is doing is "the most appropriate way" to prevent a recurrence of the tragedy.

"There is a building feeling among some on the right that this is the time to correct a policy that is fundamentally flawed, one that is too soft on the Soviet Union," one official said. "They think that this presents a real opportunity to change policies. But that's exactly the wrong thing to do at this point.

"The president understood the reality of the Soviet threat before this incident," the official continued. "This serves to confirm it, not change it. His view wasn't changed, as President Carter's was."

This was a reference to a statement made by Carter in an interview with Frank Reynolds of ABC News on Dec. 1, 1979, after the Soviet invasion of Afghanistan, which Carter said "has made a more dramatic change in my own opinion of what the Soviets' ultimate goals are than anything they've done in the previous time in office."

White House counselor Edwin Meese III said yesterday that Reagan will not make any decision on his course of action regarding the Soviets until after he meets today with congressional leaders including Senate Majority Leader Howard H. Baker Jr. (R-Tenn.), Senate Minority Leader Robert C. Byrd (D-W.Va.), House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) and House Minority Leader Robert H. Michel (R-Ill.)

An aide to Michel quoted him as saying that a majority of the House would be satisfied with "an international response," apparently similar to the one envisioned by the administration. White House officials said they also expected cooperation from Baker and the Democratic leadership despite Byrd's early call for a cancellation of the grain agreement.

"We don't see any sign that anyone is playing politics in this," said an administration official. "This isn't an issue that divides Democrats and Republicans."

But the issue has sparked renewed criticism of Reagan from restive conservatives who want the administration to use the incident as a springboard for more militant anti-Soviet actions. Sen. Jesse Helms (R-N.C.), a prominent Senate conservative now visiting Korea, said on ABC's "Nightline" on Thursday that "the president ought to expel all Soviet diplomats except the Soviet ambassador and a few functionaries." The death of Rep. Larry McDonald (D-Ga.), chairman of the John Birch Society who died aboard the Korean Air Lines plane, has added emotional fuel to the criticisms from conservatives.

Fourteen leaders of conservative groups met Friday morning at the northern Virginia home of Richard A. Viguerie, publisher of Conservative Digest and a fund-raiser for New Right causes. Many were friends of McDonald, and they agreed to hold a memorial service for him in Washington on Sept. 11.

Yesterday, Kathryn McDonald, the congressman's widow, asked Reagan to speak at the service for her husband and "the 268 other people murdered" in the shooting down of the plane. Speakes said Reagan would consider the request.

Viguerie was scathing in remarks about the president yesterday, saying the administration should have called off the arms control negotiations and canceled a recent order allowing transfer of equipment that will be used on the Soviet gas pipeline to western Europe. Even more importantly, he said, Reagan should use the incident to dramatize the importance of his defense budget increases and justify his anti-Communist policies in Central America.

"What an unbelievable opportunity to use this disaster for some good at least," Viguerie said. "It's not happening."

In the past Viguerie and his supporters often have been critical of White House chief of staff James A. Baker III and other administration officials for supposedly counseling Reagan not to follow his conservative instincts. Yesterday, Viguerie said that "maybe Ronald Reagan is really being Ronald Reagan after all" and added, "The problem is Reagan, not Jim Baker."

Administration officials said Reagan had led the way in maintaining that the arms control negotiations and the grain agreement must be maintained despite his denunciation of the Soviets.

One official said he responded to initial news of the incident by saying "that's incredible" but also expressed awareness that the Soviets had been shooting down planes which strayed over their territory for many years.

Several of the president's advisers said they believe he is likely to be credited more in the long run for a careful response that stresses cooperation with other countries and with Congress than with what one official called "an itchy-trigger response."

"The Soviet action speaks for itself, more strongly than at any time since the invasion of Afghanistan," one official said. "Reagan doesn't have to be on the bandbox saying, 'See, I told you so.' "

Reagan did speak out strongly against what he called "the outrageous Soviet attack" in his weekly radio address yesterday.

"This murder of innocent civilians is a serious international issue between the Soviet Union and civilized people everywhere who cherish individual rights and value human life," the president said. "It is up to all of us, leaders and citizens of the world, to deal with the Soviets in a calm, controlled, but absolutely firm, manner. We have joined in this call for an urgent U.N. Security Council meeting. The evidence is clear, it leaves no doubt it is time for the Soviets to account.

"The Soviet Union owes the world a fullest possible explanation and apology for their inexecusable act of brutality," Reagan said. "So far, they flunk the test. Even now they continue to distort and deny the truth."

However, Reagan stopped short of saying that the Soviets had knowingly shot down an unarmed commercial airliner. One official said it was possible that there was a mistake in the Soviet command, and another said, "We know what happened; we don't know what the motivations were."

No serious consideration was given in the administration to canceling the two sets of nuclear arms negotations with the Soviets soon scheduled to resume in Geneva. Reagan underscored the importance of the negotiations yesterday after a meeting with Paul H. Nitze, his chief negotiation at talks aimed at limiting medium-range missiles in Europe.

Nitze, who leaves today for consultations in West Germany and then goes on to Geneva for the opening of the next round of talks on Sept. 6, said that the airplane incident "certainly isn't going to help progress" at the talks but that "we must nevertheless continue our efforts to reduce the threat of nuclear conflicts through negotiated, fair and verifiable agreements."

Officials said the Soviets already are paying a price for shooting down the airliner because of the animosity the Soviet action has engendered in western Europe.

"The idea that we might have to 'reach' to get an agreeement" is no longer valid, one administration official said. His point was that the West was less vulnerable to demands from internal critics or the Soviets for concessions at the arms talks.

Nitze said, "It will take greater cooperation on their the Soviet side to get a deal" at the negotiations.

In explaining Shultz's attendance at the 35-nation Madrid conference on European security and human rights, where he will meet with Gromyko, one official said it is "important to hold the Soviet Union accountable for the agreements and commitments they make." The conference has reviewed compliance with the 1975 Helsinki agreements on security and human rights.

Staff writers Helen Dewar and George C. Wilson and researcher Maralee Schwartz contributed to this report.