Foreign Minister Yitzhak Shamir today reacted cautiously but did not reject outright President Reagan's offer of U.S. guarantees for the security of Israel's northern border after all Israeli troops withdraw from Lebanon.

"We are not asking now for American guarantees," Shamir said in a radio interview. "We want to have an agreement between us and Lebanon."

Asserting that Israel's security "must be based on close cooperation between the Lebanese and Israeli forces," Shamir said, "it may be that American guarantees could be helpful, but they couldn't replace an agreement between us and Lebanon."

Reagan made the proposal in a foreign policy speech to the American Legion in Washington. He did not specify what guarantees of security the United States was prepared to offer, but Shamir reiterated Israel's objections to the deployment of outside forces along the border.

"We don't think the presence of foreign forces, multinational or international, could be helpful," he said. "It is not for them to enable us to be assured that the PLO Palestine Liberation Organization will not come back to the frontier. They will not fight against them.

"The only parties who are able to do it, who have the motivation to do it, are Lebanon and Israel."

There was no other immediate Israeli reaction to the Reagan speech. A spokesman for Prime Minister Menachem Begin said Begin was waiting to see the full text of the speech.

Earlier today, Shamir also provided the first Israeli reaction to the Palestine National Council meeting in Algiers, declaring that the council's deliberations "will not change anything regarding events in this region and will not contribute a thing to peace."

Shamir called the meeting of the PLO's parliament-in-exile a "deception and a fraud."

"They the PLO will not return to Jerusalem with the aid and sponsorship of Jordan's King Hussein," Shamir told the Jerusalem Economic Club.

"Israel has declared that it is prepared to conduct negotiations with the king of Jordan only in the framework of Camp David peace agreement , and not in order to return Jerusalem to the PLO. If this council has something to contribute to peace, it has to decide only one thing. That is to dissolve itself and call on the Palestinian Arabs to take the only way open to them, the path of cooperation with Israel within the framework of Camp David."

The full council approved a resolution that stopped short of outright rejection of Reagan's Middle East peace initiative, which calls for negotiations leading to the linking of much of the Israeli-occupied West Bank and Gaza Strip to Jordan. U.S. officials hope such a stance by the council will encourage Hussein to join peace negotiations based on the Reagan plan.

Israel rejected the Reagan plan immediately after it was announced by the president last September. The Israeli government says it is willing to negotiate with Hussein, but only on the basis of the Camp David formula, which calls for an interim five-year period of autonomy for the residents of the West Bank and Gaza Strip before the final status of the territory is determined.

The Begin government has declared repeatedly that it will not surrender sovereignty over the West Bank, which it considers a part of historic Israel.

The Israelis are insisting in the troop withdrawal talks with the Lebanese on the right to leave several hundred soldiers at camps in southern Lebanon even after all other foreign forces have left the country. In his speech to the economic club, Shamir sounded an optimistic note on the negotiations, the slow pace of which has been a source of friction between the United States and Israel.

Shamir gave no reasons for his optimistic projection, but Israeli press accounts here today also reported progress in the talks. The independent newspaper Haaretz said Israel had scaled down its demands for "observation posts" in Lebanon in return for the Lebanese softening their objections to the continued deployment of forces commanded by renegade Lebanese Army major Saad Haddad, which are supported by Israel, in the southern part of the country.