Israeli Foreign Minister Yitzhak Shamir sought yesterday to persuade Secretary of State George P. Shultz that Israel's security requires close Israeli-Lebanese military cooperation and possibly Israeli observation posts in southern Lebanon after withdrawal of Israeli forces there.

Shamir made his pitch during a four-hour State Department meeting, which Israeli sources described as an attempt to explain at the "highest possible levels of the U.S. government" the Jewish state's ideas about future security arrangements and, if possible, to win U.S. backing for Israel's position.

Officials of both nations said that, while the session was characterized by "great cordiality" and frequent expressions of understanding by Shultz for Israel's concern, the secretary concentrated on listening and gave no indication of the U.S. response.

That is expected to come at a second lengthy meeting today. Although U.S. officials were cautious about predicting what Shultz will say, the expectation is that essentially he will reiterate the U.S. view that Lebanese President Amin Gemayel cannot accept a continuing Israeli military presence in his country and that the security problem should be resolved by placing a multinational force, probably including U.S. troops, in southern Lebanon.

While it is not on Shamir's schedule, there were strong suggestions last night that he will meet with President Reagan later today or tomorrow. If such a meeting takes place, Reagan is expected to underscore U.S. concern for Israel's security while reaffirming the administration's position that the problem can be handled in ways that do not require continued Israeli presence in Lebanon.

Shamir, who spoke briefly with reporters after the meeting, said he did not want to discuss specifics of the Israeli proposal. But he signaled the general outlines of what Prime Minister Menachem Begin's government wants when he said:

"It is clear to us that in the coming month--the near future--the Lebanese government will not be able to control the security of its territory by its own forces. Therefore, it's our conviction that there must be close cooperation between us and Lebanon to achieve these security goals."

Sources familiar with the talks said Shamir's presentation consisted essentially of variations on ideas Israel has put forward during the 10 weeks of stalemated negotiations for a plan to achieve withdrawal of Israeli, Syrian and Palestine Liberation Organization forces from Lebanon.

In addition to establishing a security zone in parts of Lebanon bordering Israel, the Israelis have called for early-warning observation posts to be manned by Israeli troops, joint Israeli-Lebanese patrols and a major role for the Lebanese Christian militia of Saad Haddad.

Haddad is a cashiered army major allied with Israel but regarded as a threat to Gemayel's authority.

The Gemayel government, with U.S. backing, has rejected Israel's security demands and its call for normalized relations as unacceptable to the factions he is seeking to unify within Lebanon, and as unacceptable also to Arab countries whose financial and political support he needs.

The positions conveyed by Shamir actually represent a softening of earlier Israeli demands. Ouster of Ariel Sharon, most ardent advocate of a tough line on Lebanon, as defense minister has meant a shift of influence within the cabinet to Sharon's successor, Moshe Arens, and Shamir, who reportedly want an accommodation with Washington.

Despite rumors of an impending agreement, Shultz insisted in an interview with The Washington Post Saturday that no deals were made in advance. Instead, the Shamir visit is seen here as an attempt to appeal to Shultz over the head of Reagan's special Mideast envoy, Philip C. Habib, whom the Israelis regard as unsympathetic to their views, and to gauge how much flexibility Washington is willing to show toward the modified Israeli positions.

But, at Shultz' insistence, Habib and the other U.S. special envoy, Morris Draper, are participating in the talks here in what U.S. officials say is a deliberate signal that they speak for the United States and that Israel must deal with them.

Shultz also invited a Lebanese delegation that now includes Foreign Minister Elie Salem and ex-prime minister Saeb Salam to be present during Shamir's visit to enable the United States to ascertain Lebanon's views through separate talks.

These moves reportedly have dampened Israel's hope that Shamir could win Shultz' assent to the Israeli proposals and make a deal that would short-circuit the negotiating process. Shamir told reporters that idea was "completely baseless," denied his government is unhappy with Habib and said he has no objection to the presence of the Lebanese, who are to meet separately with Shultz today.