Two weeks after Israeli Foreign Minister Yitzhak Shamir's visit here raised hopes for early agreement on withdrawal of foreign forces from Lebanon the negotiations have become deadlocked again by what U.S. officials say are unacceptable Israeli demands for security arrangements in southern Lebanon.

"We are one step away from being back to square one," one U.S. official familiar with the negotiations said yesterday in describing "a progressively negative" Israeli reaction to the proposals for ensuring the security of Israel's northern border made to Shamir by Secretary of State George P. Shultz.

Shamir, who conferred with Shultz for three days in mid-March, returned home sounding optimistic about the chances for a breakthrough. But U.S. officials said yesterday that in subsequent negotiations with President Reagan's special Mideast envoy, Philip C. Habib, the Israelis have rejected the idea that Lebanese forces can control terrorist activity in southern Lebanon. And they have reiterated demands for arrangements unacceptable to Lebanese President Amin Gemayel.

Habib is expected to return here today. He is coming back partly for personal business, but the officials said he also is expected to discuss with Reagan and Shultz possible new U.S. steps for breaking the latest impasse.

The most immediate problem, according to U.S. officials and to reports from Jerusalem, involves Israel's insistence that the Lebanese Christian militia forces of Saad Haddad, a rebel Lebanese army major allied to Israel, continue to have a largely autonomous role in controlling a southern Lebanon security zone.

Gemayel, supported by the United States, has insisted that Haddad step aside and put his forces under control of the central government.

In addition, U.S. officials said, Prime Minister Menachem Begin's government still insists on the right to maintain some kind of residual military presence in southern Lebanon after the main force of Israeli troops is withdrawn.

The United States strongly opposes that idea because it would undermine the efforts of Gemayel, who is a Christian, to gain support for his fledgling government from Lebanese Moslems and because it would be unacceptable to Syria, which has an estimated 35,000 troops in northern and eastern Lebanon. According to U.S. officials, Syria has made clear to Habib that it will not pull out its troops and Palestine Liberation Organization forces under Syrian protection unless there is a total Israeli withdrawal from the south.

Although Israel's attitude in recent days is known to have caused great frustration and irritation among U.S. policy makers, the officials stressed that the administration still has not decided whether to take a tougher stance toward Begin.

Some U.S. policy analysts say they believe the Israeli position may be prompted by the Begin cabinet's desire for concessions to protect itself against charges of bowing to U.S. pressure. If that analysis is correct, U.S. officials said, it should be possible through further negotiation to work out a formula about Haddad's future and other matters that will be acceptable to Begin. However, the officials added, there also is a growing feeling here that the Israelis, despite their publicly stated position of wanting only to ensure their territory against terrorist attacks, have as their real aim the continued de facto partition of Lebanon, with the south controlled by Israeli surrogates such as Haddad.

Even if that is not the case, the officials stressed, the U.S. view is that the terms being insisted upon by Israel make it impossible to achieve an accord that can be accepted by Gemayel and Syria.

As a result, the officials added, unless Israel moderates its present demands substantially and agrees to bargain on the basis of the security arrangements proposed by Shultz, there seems to be little chance of avoiding a major new confrontation between Washington and Jerusalem.