Israeli Foreign Minister Yitzhak Shamir met with Secretary of State George P. Shultz yesterday to discuss Israel's proposals for withdrawing its forces from Lebanon under conditions that the United States fears will give Israel continued effective military control over southern Lebanon.

On the American side, there were hints that the Reagan administration, as an alternative, might consider sending U.S. troops to southern Lebanon as part of a multinational peace-keeping force similar to the one that has American Marines in Beirut.

U.S. and Israeli sources both said the idea of using such a force to police a security zone in southern Lebanon was not discussed in the four-hour meeting between Shamir and Shultz. However, a hint that U.S. officials might be thinking along those lines came in remarks yesterday by Paul Wolfowitz, head of the State Department policy planning staff, to the Washington Press Club.

Wolfowitz said he "could well envision arrangements . . . where something like UNIFIL the United Nations International Force in Lebanon or some other multinational force would provide a function in southern Lebanon in giving Israel some assurance that events there wouldn't affect Israel's security. . . ."

He did not mention U.S. participation or give other specifics. However, other U.S. officials, who asked not to be identified, noted that Israel will not accept UNIFIL as the force to patrol a security zone across its northern borders. As a result, these officials added, if the United States wants a speedy Israeli withdrawal from all of Lebanon, Washington might be forced to consider creating a new multinational force with U.S. participation as "the option of last resort."

According to one of these officials, that option "hasn't forced itself on the agenda yet." But he noted that the Israeli cabinet's withdrawal proposals, which Shamir presented to Shultz yesterday, contain ideas about a southern Lebanon security zone that are a problem for the United States and that will force Washington to propose some other solution if it is unwilling to accept the Israeli plan.

Specifically, the Israelis are calling for creation of a security zone extending 25 to 35 miles north of Israel's borders with Lebanon, with the job of policing the area to ensure against terrorist infiltration being turned over to the Lebanese army.

The problem, from the U.S. point of view, is that the Lebanese army, which is in disarray after years of civil war, will require a long period of retraining and restructuring before it can perform that task effectively. The Israelis are known to be arguing that this problem can be overcome by assigning a major role in the policing operation to the Israeli-backed Christian militia forces headed by Saad Hadad, a renegade Lebanese army major.

However, that idea is opposed by both the new Lebanese government of President Amin Gemayel and the United States. They see a continued role for Hadad as a ploy to allow Israel to maintain control over southern Lebanon through him, and as a major impediment to reestablishment of a strong central government able to exert its authority through all of Lebanon.

Initially, the United States had hoped to resolve the security-zone problem by giving UNIFIL more troops and a broadened mandate and assigning it the task of policing the zone. But that idea is anathema to Israeli Prime Minister Menachem Begin's government, which regards UNIFIL as ineffective and subject to the pressures of the Arab countries and their Third World allies in the United Nations.

That is what has given rise to the tentative stirrings in U.S. policy-making circles about the idea of a possible new multinational force. However, there would be no hope of Israel accepting that idea either without U.S. military participation; and pursuing that route would confront the administration with some potentially serious problems.

Such a move could stir heavy opposition from Congress, where many members have expressed qualms about the presence of American Marines in Beirut and have warned against a deepening U.S. involvement in Lebanon. In addition, Defense Secretary Caspar W. Weinberger and the Joint Chiefs of Staff, who are unhappy about having the Marines in a potential casualty-producing situation, could also be expected to oppose sending more American forces to other parts of Lebanon.

For the moment, these factors seem likely to make the administration move very slowly and cautiously on the multinational force idea. White House spokesman Larry Speakes said yesterday he knew of no plans to send U.S. forces to southern Lebanon, and Wolfowitz stressed that, while he was suggesting the possibility of a new multinational force because of Israel's hostility to UNIFIL, the U.S. preference is "to make use of UNIFIL because it's there."

In the meantime, it was clear from yesterday's meeting between Shultz and Shamir that arduous and complicated negotiations will be required to work out the problem of getting foreign forces -- some 70,000 Israelis, 25,000 Syrians and 10,000 Palestine Liberation Organization fighters -- out of Lebanon.

The session did result in agreement to create a working group, which began meeting last night and which sources on both sides said was charged with trying to spell out the differences in the various plans and find possible areas of compromise on which to base future discussions.

The American team in the working group is being headed by Deputy Secretary of State Kenneth W. Dam, who has been put in charge of the administration's interagency work on the Lebanon situation. Israeli sources said Shultz and Shamir had agreed tentatively to meet again, probably this weekend, to review the results of the working group's efforts.

U.S. sources said the Shamir visit and another by Gemayel, who will be received by President Reagan on Tuesday, are being viewed here as a "first-cut, high-level attempt" to bridge some of the differences in the way of a withdrawal plan that will be acceptable to the various sides in the Lebanon conflict.

The hope, the sources continued, is that they will provide a springboard for Reagan's special envoy, Morris Draper, to return to the Middle East and accelerate his efforts to bring Israel, Syria and the Lebanese government into agreement on a workable plan that will meet the hoped-for timetable of withdrawal by the end of the year.

Shamir is to meet today with Vice President Bush and Weinberger. However, Israeli sources said these sessions will be focused more on the general state of U.S.-Israeli relations than on the specifics of the Lebanon situation.

In addition to Lebanon, Shultz and Shamir discussed the overall Mideast situation, including Reagan's initiative for resolving the Arab-Israeli conflict, which has been rejected by the Begin government.

The indications were that the discussion was very general, with no new ground being broken. However, Shamir, who spoke briefly with reporters, said the talks were "conducted in a spirit of friendly consultations," and Israeli sources went out of their way to stress that they regarded the meeting as "the renewal of a friendly dialogue" between Washington and Jerusalem following the strains of the Lebanon war and differences over the Reagan initiative.