The Reagan administration changed its tactics yesterday in dealing with the hijacking of Trans World Airlines Flight 847, deliberately playing down diplomatic efforts to free the 40 American hostages in Beirut and emphasizing that no quick solution appears to be in sight.

White House and State Department officials referred to the situation as "static" or "frozen" and placed tight limits on the manuevering room that the United States is allowing third parties attempting to mediate the crisis.

President Reagan, who held an emotional private meeting with family members of three hostages in Dallas, said the United States would "continue to act with appropriate restraint" but pledged that terrorists "will never succeed in weakening our resolve."

Meanwhile, State Department spokesman Bernard Kalb said that any assistance provided by other countries in an attempt to free the hostages "must fall within the context of the fixed American position" of refusing to make concessions to the hijackers to win the release of the hostages.

While emphasizing that the United States would not pressure Israel to make any decisions, some U.S. officials expressed the hope that the Israelis may decide on their own to free more than 700 Lebanese prisoners, most of them Shiites, who were captured during Israel's withdrawal from Lebanon. The hijackers and Nabih Berri, the leader of the Shiite Amal movement who claims to have custody of most of the hostages, have demanded that the Lebanese prisoners be turned loose before the hostages are freed.

While hopeful that Israeli actions and Berri's control of the prisoners will lead to what one source called "a natural conclusion to the crisis," some sources said the sticking point is likely to be the control exercised over five or six prisoners by a radical Shiite faction known as Hezbollah, or Party of God. Informed sources said Wednesday that this group of hostages consists of government or military personnel, not persons with "Jewish-sounding names" as first believed.

Kalb yesterday said that Berri "cannot escape the responsibility" for these hostages, adding, "We and many others have made it clear to him that we consider him responsible for the proper treatment and safe return of all the hostages and the prompt resolution of this affair."

One U.S. official said of the current approach to the problem, "We're not trying to influence Israeli decisions one way or the other -- they'll do what they want at a time and place of their own choosing. They've kept their options open."

One option would be for the Israelis to stay with their pre-hijack intention of releasing the Lebanese prisoners in stages. This clearly would be welcomed by the U.S. government, which on April 3 described the taking of these prisoners as a violation of international law.

But U.S. officials, who earlier in the week indirectly pressured Israel by repeatedly referring to the U.S. view that the Lebanese prisoners were held illegally, yesterday refused to make any critical comment about Israeli actions. Administration sources indicated they may have to reevaluate their approach after the weekend.

The Israeli cabinet will meet again Sunday. Last Sunday the cabinet said it would free the Lebanese prisoners, but only if the United States publicly requested it. Reagan has said he will not make any such request, on the grounds this would be asking for Israel to make a concession to the hijackers that the United States is unwilling to make itself.

The administration's low-key approach to the week-old crisis represented what one official called "an attempt to let events take their course by lowering the spotlight." His suggestion seemed to be that it has been difficult for Israel to release the Lebanese prisoners as long as this action seemed to be in response to pressure from the United States or its allies.

The official declined to say how long the United States would stay with this approach if it produced no results. Other officials said that there was already what one called "the stirrings of a debate" within the administration over a long-term strategy if the present situation becomes an impasse.

The president's contribution to the low-key approach was to meet privately with the hostage families. White House spokesman Larry Speakes said he reassured them "that our paramount interest is the safety of those being held."

In contrast to this private meeting, Reagan will fly from Camp David today to Andrews Air Force Base for a high-visibility ceremony marking the return of the bodies of four Marines held in a terrorist attack in El Salvador. One White House official said there is "considerably more sentiment" for retaliation against this terrorist attack because there are no Americans being held hostage in Central America.

One prominent Republican, House Minority Whip Trent Lott (Miss.), said yesterday that he interpreted Speakes' statement Thursday that "our limits have been reached on terrorism" to be a commitment to at least "try to find a way" to retaliate against those responsible for the hijacking once the hostages have been freed.

Lott said that some Republican members of Congress, particularly in the South and West, are being pressured by frustrated constituents who want "retribution" for terrorist acts. He said he is working with younger House members to draw up a resolution "stating our concern, our disgust."

But Reagan has been told by his advisers that it would be difficult to retaliate in Lebanon without killing innocent people, a course of action that the president said at his news conference Tuesday would be "an act of terrorism itself."

Reagan received support for his "wait-it-out" strategy in dealing with the hostages yesterday from his predecessor, President Jimmy Carter. In a statement issued from Plains, Ga., Carter urged citizens to give their "full support and encouragement" to the president.

"I know from personal experience how difficult it is to deal with the kidnaping and prolonged holding of innocent Americans," Carter said. "Under such circumstances, our leaders have a dual responsibility: to guard the honor and interests of our nation and to protect the lives and safety of the hostages while doing our utmost to secure their early release. With patience and sound judgment these two obligations need not be in conflict."

Carter's presidency and his reelection campaign against Reagan in 1980 were severely damaged by the 444-day Iranian hostage crisis. At the time Reagan used this crisis as a symbol of what he sometimes called "a failed presidency."

The Iranians freed the 52 American hostages on Reagan's Inauguration Day, Jan. 20, 1981.