Special Mideast envoy Philip C. Habib returned here last night after failing to make measurable progress on arranging withdrawal of Israeli forces from Lebanon, setting the stage for a new policy review by high officials of the disappointed Reagan administration.

A senior State Department official declined to say what the administration will do to break the stalemate. But he made no secret of the administration's unhappiness, saying:

"The fact is that foreign forces are in Lebanon, and we don't have any plan at hand to get them out . . . . We're disappointed. We had hoped for a lot more progress than has been made."

The senior official briefed State Department reporters on the basis that he not be identified.

In private, several administration officials denied a published report that President Reagan is prepared to cut off arms aid to Israel if Prime Minister Menachem Begin's government continues to be inflexible.

Publicly, however, administration spokesmen were less definite in their denials, suggesting a reluctance to reassure Israel completely.

Habib's return, coupled with a meeting scheduled here Thursday between Reagan and Egyptian President Hosni Mubarak, marks an intensification of U.S. efforts to achieve progress toward the dual goals of a withdrawal from Lebanon and broadening the Middle East peace process.

Reporters were told that it is likely that Habib will return to the region shortly after the talks with Mubarak are concluded this week, and the senior State Department official held out the possibility that Secretary of State George P. Shultz might travel to the area soon.

Compounding frustration at the slow pace of the Lebanon talks has been the increasing concern of U.S. military leaders about further new and serious confrontations between the U.S. Marines and other forces in Beirut as the U.S. military presence there lengthens.

Withdrawal of foreign forces from Lebanon also affects the wider question of whether King Hussein of Jordan can be induced to join the peace initiative set forth by Reagan last Sept. 1.

There has been recurrent speculation that Hussein, working on secret U.S. assurances, early in March will announce his willingness to join negotiations on the future of Israeli-occupied Arab territories.

However, the senior official who briefed reporters was unable to report any decisions by Hussein, and indicated that much depends on the elaborate process of consultation under way in the Arab world.

Some headway has been made toward broadening the peace process, the official said, but "obviously not the breakthroughs we had hoped."

Over the next six weeks, the sequence of events now unfolding could be decisive to the success or failure of U.S. efforts in Lebanon and the overall peace process, according to several officials. But they also assert that much of the speculation about the situation is an inaccurate representation of what is happening.

On the Lebanese negotiations, these officials say, the administration, while increasingly annoyed by what it regards as unreasonable Israeli demands, is not thinking in terms of action as drastic as an aid cutoff. That prospect was raised yesterday by syndicated columnists Rowland Evans and Robert Novak.

One official, who asked not to be identified, said: "It's not totally beyond the realm of comprehension that we might have to consider measures of that sort. But the idea of moving to such a step is still a very long way down the road, and the current thinking is that this can be worked out before we get anywhere near that area."

At the same time, these officials stress that the administration believes it is necessary to break the deadlock over Lebanon.

Several sources said Habib, ostensibly recalled to participate in talks with Mubarak, actually is here to discuss ways of persuading the Israelis to modify demands that the United States believes are politically impossible for the weak government of Lebanon to accept.

Begin's desire for a meeting next month with Reagan is being held hostage by the administration to progress in the Lebanon talks.

While further steps will not be decided until Habib's report has been delivered, the likely U.S. strategy, according to the sources, will be to continue putting off a date for a visit by Begin, while publicly becoming more vocal in assigning blame for the delay and any increased tensions in U.S.-Israeli relations to the Begin government's attitudes.

In line with that approach, the administration left some ambiguity in its public statements yesterday about the accuracy of the Evans-Novak report. When White House deputy press secretary Larry Speakes appeared to be edging toward a firm denial of an impending aid cutoff, another White House spokesman, Mort Allin, interjected: "We are extremely concerned at the slow pace of the negotiations."

On the related question of the U.S military presence in Beirut, senior officials acknowledge that the Defense Department has been uneasy about the Marines' role there. But, they add, the president has concluded that Marines' presence is required to bolster Lebanese government authority and has not wavered from his decision that, despite the risks, the Marines must remain if the high-priority goal of stabilizing Lebanon is to be realized.

In addition, the officials said, current thinking within the administration is that the size and mission of the force must be increased eventually. The United States has made no commitments to do so, but the delay, according to these sources, is due to the administration's insistence on knowing more about what will be required after a withdrawal agreement.

With regard to Hussein, who conferred with Reagan here last month, the officials say that they have encouraging, if not specific, indications that he is making progress toward getting approval of the Palestine Liberation Organization and Saudi Arabia to enter broadened peace talks. Hussein is known to have told Reagan that it would be virtually impossible for him to move without such approval.

The officials add, however, that while the United States urged Hussein in the strongest possible terms to come forward, he was given no assurance that Washington, in turn, would seek to force Israel to freeze settlements in the occupied territories or guarantee how much territory would be returned to Jordanian control.

The senior State Department official, reiterating a position previously stated in public by Shultz, declared emphatically that Israeli settlements are "not constructive" and added, "A lot of things going on in the West Bank are not constructive."

As to the future of Jewish settlements, he indicated that Jews would be able to continue living on the West Bank after a peace agreement, but subject to the area's governing authority.

The meetings with Mubarak are expected to involve bilateral issues, including Egypt's requests for increased U.S. military and economic aid. U.S. officials also want to urge Mubarak to improve Egypt's cool relations with Israel, in order to reassure the Jewish state that its peace treaty with Egypt is durable.