Israeli Foreign Minister Yitzhak Shamir's talks with Secretary of State George P. Shultz ended yesterday, with the United States refusing to accept Israel's bid to keep a military presence in southern Lebanon as part of an agreement to withdraw Israeli and other foreign forces from that country.

Speaking with reporters after what he called "two days of exhausting talks," Shamir acknowledged that, despite his efforts to win Shultz over, "the Americans do not accept the principle of an Israeli presence" in southern Lebanon.

Shamir insisted that there was complete accord between the two governments on the principles that should govern a solution of the thorny Lebanon problem. But, by adding that there is still no agreement on the "modalities and form" for implementing these principles, the minister, in effect, conceded that he had failed in his attempt to reverse the U.S. argument that Israel's security needs can be met through measures that do not breach Lebanese sovereignty.

Left unclear was whether the 10-week-old withdrawal negotiations are headed for further prolonged stalemate or whether Israeli Prime Minister Menachem Begin's government, having determined that the United States will not budge, now will modify its security-arrangement demands in the interest of averting new tensions with the Reagan administration.

Shamir hinted that some Israeli flexibility might become evident before too much longer. He said that both sides "now have a better understanding of each other's positions" and that "this understanding will find its expression in the negotiations that follow without delay." He added:

"I believe that we will get an agreement. I cannot say if it's very near. But we will get an agreement. We Israel have no interest to stay in Lebanon."

In the view of U.S. officials, Shamir's trip here represented an attempt to go over the head of President Reagan's special Mideast envoy, Philip C. Habib, whom the Israelis regard as unsympathetic to their views, and appeal directly to Shultz for greater understanding of their desire for an Israeli presence in southern Lebanon to deter cross-border terrorist attacks.

Shultz pointedly insisted that Habib and the other U.S. negotiator, Morris Draper, be present. He also invited a Lebanese delegation headed by Foreign Minister Elie Salem to come to Washington so that U.S. officials could get the opinions of President Amin Gemayel's government on the Israeli proposals.

Both moves were intended as signals to the Israelis that Habib speaks for Reagan and that the United States has a strong interest in preventing any action by Israel that might undermine Gemayel's efforts to consolidate his authority within Lebanon and to win the backing of other Arab goverments.

At the same time, according to U.S. officials, Shultz wanted to reassure Shamir that the United States understands Israel's security concerns and is willing to consider measures in southern Lebanon that do not involve infringements of Lebanese sovereignty.

Shamir acknowledged that he had been treated with "great courtesy and understanding" both in his talks with Shultz and in a half-hour meeting with Reagan at the White House yesterday.

He said the president had told him that a way must be found to reconcile Lebanese sovereignty with Israeli security, and he quoted Reagan as saying:

"It's time that we stopped talking about each other and started talking to each other."

That was an apparent reference to the strains caused in U.S.-Israeli relations by the impasse in the withdrawal talks. It also appeared to be an attempt by Reagan to encourage signs that Shamir and the new Israeli defense minister, Moshe Arens, want to resolve the Lebanon issue before it causes potentially irreparable strains in the relationship.

Some U.S. officials have even speculated that Shamir's trip was intended, in part, to demonstrate to hard-liners in the Begin government that the United States will not give ground over Lebanon and to strengthen the hand of those in Israel who want a more flexible approach.

Whether that will prove to be the case won't become clear until the negotiations proceed further. And Shamir, despite his hints yesterday about flexibility, also reiterated Israel's insistence that close Israeli-Lebanese military cooperation, involving some form of Israeli presence in southern Lebanon, will be required for some time to come.

In the past, the Israelis have talked about Israeli-manned observation posts and joint Israeli-Lebanese patrols in the areas.

Yesterday Shamir declined to discuss specific measures and said: "If you're talking about an occupation, we don't ask for that. For us, the principle of close cooperation for an agreed time is what's important."

He also reiterated Israel's objections to the idea of the security problem being entrusted to a multinational force, including U.S. troops, in southern Lebanon.

He said: "We are opposed to the presence of any foreign forces in this zone, American or multinational. We don't believe it will contribute to the security of the zone."