The Reagan administration maintained a firm line yesterday against making concessions to the hijackers of Trans World Airlines Flight 847, while U.S. officials struggled to find a diplomatic opening that could lead to the release of the 40 American passengers and crew members being held hostage in Beirut.

"We're pressing all the buttons and hoping that eventually one of them will light up," a senior administration official said.

Campaigning for his tax-overhaul plan in Indiana, President Reagan said, "We are continuing to do everything we can to bring all credible influence to bear to get our people freed and returned home safe and sound. But let me say we must not yield to the terrorist demands that invite more terrorism. We cannot reward their grisly deeds. We will not cave in."

Secretary of State George P. Shultz, testifying before a Senate committee, spoke of an "unwavering" determination to resolve the crisis and said that conceding to the hijackers would be "tantamount to succumbing to blackmail."

But administration officials acknowledged that there did not appear to be signs of progress in their efforts to free the Americans, most of them now under the custody of Nabih Berri, leader of the Shiite Amal movement.

White House chief of staff Donald T. Regan, returning from Indianapolis aboard Air Force One, said there had been no change in the previous 24 hours and no reply from Berri to Reagan's reiteration at his news conference Tuesday night that the United States would refuse to pressure Israel to release more than 700 prisoners captured during its withdrawal from Lebanon.

Berri has said these prisoners must be released before the hostages will be freed, but administration officials are insistent that the United States will not apply pressure on Israel to accomplish this. White House spokesman Larry Speakes said there is no "unsaid, unspoken deal" to prompt Israel to release the prisoners.

"He isn't going to get the Shiites out of prison by holding these people who are innocent," one administration official said of Berri. "He'll have to find a way to de-link the hostages and let nature take its course. It's not our problem, it's his problem."

Administration officials denied repeatedly that the United States is urging the International Committee of the Red Cross to arrange a trade of the Israeli prisoners for the hostages.

"The notion is about, and it is flat wrong, that the United States is urging behind the scenes some arrangement that the Red Cross could broker some kind of phased movement" of the prisoners, said national security affairs adviser Robert C. McFarlane, who repeated Reagan's words of Tuesday night that "we will not make concessions and will not ask anyone else to do so."

Diplomatic sources disclosed yesterday, however, that the International Red Cross was involved in an attempt to negotiate the Americans' release while the TWA plane was in Algiers Saturday morning.

U.S. diplomatic efforts to free the hostages appeared yesterday to be focused in at least three directions.

The first was to keep pressure on Berri, who Reagan said in his news conference would be held responsible if the hostages were handed back to the hijackers.

The second initiative involved using the good offices of Algeria, whose efforts to end the crisis have repeatedly been praised by U.S. officials. Algeria's effort is being led by its ambassador in Beirut, Abdelkrim Kraieb, who on Monday cut short a visit to Paris to meet with Lebanese government officials and senior Shiite Moslems.

The United States also was seeking to involve Syria in the discussions. A White House official said Syrian President Hafez Assad may have played "a constructive role" in involving Berri in the negotiations after Reagan cabled Assad on Friday, asking for assistance.

The official confirmed that a second cable asking for further assistance had been sent to Assad but the official declined to reveal details of the message or whether there had been a response.

The varied nature of the diplomatic approaches reflected the difficulty the administration is encountering in trying to find a way out of the crisis. Richard W. Murphy, assistant secretary of state for Near Eastern affairs, told a House committee that "we're trying to establish a network of contacts" to Berri.

But officials said this effort was in a preliminary state. "Right now it's not even a slow dance," one official said. "The parties haven't even gotten onto the floor yet. They're still inching their way to the edge."

An additional complication was revealed yesterday by McFarlane, who said that an extremist Shiite Moslem group known as Hezbollah is holding "half a dozen passengers with Jewish-sounding names" who were removed from Flight 847 before the rest of the passengers were taken off the plane.

Hezbollah, which means Party of God, is a terrorist organization associated with a faction of pro-Iranian Lebanese Shiites known as Islamic Amal, which split from Berri's mainstream Amal movement a few years ago. Hezbollah is generally regarded as responsible for the 1983 bombing attacks against the U.S. Embassy and the Marine compound in Lebanon.

Meanwhile, the Rev. Jesse L. Jackson talked with State Department officials and said he was willing to go to Lebanon in an attempt to free the American hostages but would not do so without a prior understanding that Israel would free the Shiite prisoners.

The administration also heard from Jordan's King Hussein, who telephoned Reagan Tuesday night to denounce the hijacking as "a dastardly crime," according to Shultz.

In Reagan's news conference, he asked U.S. citizens not to travel to Middle Eastern countries that did not condemn the hijacking. Speakes said Reagan has since talked to several heads of state in the region and received "positive responses" from most of them. He specifically named Jordan, Israel and Tunisia as countries that had publicly denounced he hijacking.

Responding to Reagan's appeal to U.S. airlines to consider suspending flights to Athens, Pan American World Airways yesterday suspended its daily flight to that city. Reagan criticized Greece for failing to provide adequate security at the Athens airport, from which Flight 847 was hijacked Friday.

The president, sticking to his schedule despite the crisis, paused during his speechmaking in Indiana to meet with the family of James Hoskins, 22, one of the hostages.

The president also answered questions from reporters during a stop at Mac's Family Restaurant in Mooresville, Ind., saying that "we're doing everything we can to put the pressure on and bring these people home safely." Speaking of the hostages, Reagan said: "I've been praying ceaselessly for them and for their safety."

But administration officials refused to set any timetable on their release or to speculate on the success of diplomatic efforts. On the flight home, chief of staff Regan said the administration was "playing one day at a time, keeping all options open."