Israeli Prime Minister Yitzhak Shamir has sent an envoy here to tell Secretary of State George P. Shultz that President Reagan's "territory for peace" principle cannot serve as the basis for Israel's entering new peace negotiations on the fate of the strife-torn occupied West Bank and Gaza Strip.

The envoy, Ehud Olmert, a member of Shamir's Likud coalition in the Knesset, met Sunday with Shultz's executive assistant, Charles Hill, to discuss the proposed new American approach to the Middle East peace process.

The approach envisages initial Arab-Israeli talks on limited autonomy for the occupied territories and then negotiations on their final status, starting within six months, based on the "territory for peace" principle contained in President Reagan's September 1982 speech on the Middle East.

Radio Israel reported that Olmert told Hill that Shamir would "never" accept to negotiate a final settlement for the territories on the basis of this formula and warned that if Shultz continues with this approach, the new American peace initiative is doomed to failure.

Shultz recently told visitors to the State Department that all parties must clearly understand that the new U.S. peace initiative will go forward only on the basis of the "territory for peace" formula and U.N. Resolution 242 guaranteeing peace and security for all Middle Eastern nations.

An Israeli Embassy spokesman confirmed that Olmert met Hill Sunday "for a discussion of the American initiative." He described it as "an important meeting" but said it was "not ground-breaking."

The spokesman, Yossi Gal, said Olmert had been in Palm Beach, Fla., for a meeting of the B'nai B'rith Anti-Defamation League, where Shultz was the guest speaker Friday. Olmert reportedly met Shultz briefly there but, because he did not have time to discuss the new American initiative, Olmert came here Sunday to see Hill.

Yesterday Shamir, leaving Israel for a visit to Italy, publicly stated his opposition to any negotiations based on the "territories for peace" formula. A State Department official said there would be no comment on Shamir's statements or the visit of his latest envoy.

More than 50 Palestinians have been killed and thousands arrested by Israeli authorities in the territorities in the past two months. The violence has spurred the Reagan administration after months of diplomatic inactivity to search for ways to revive the long-stalled peace process.

Shamir appears to be hardening his position on the issue of peace talks in preparation for Shultz's visit to the region to discuss the new American peace approach starting Feb. 24. Shultz is expected to meet with Shamir Feb. 28-29, after visiting Egypt and Jordan.

Shamir's mounting opposition to opening negotiations on anything other than limited Palestinian autonomy at this time suggests the new U.S. peace initiative calling for final-settlement negotiations as well may be headed for difficulties.

However, Shultz appears to have found an unexpected ally in the American Jewish community, which has been extremely supportive of his new approach and is pressing Shamir to show flexibility. A delegation of 60 Jewish leaders is planning to visit Jerusalem for talks with Shamir at the same time Shultz is there, according to Radio Israel.

The Israeli prime minister is also pressing Shultz to avoid spelling out any American position on the outcome of the final status negotiations ahead of time and to remain noncommittal, according to Israeli sources.

Reagan, in a September 1982 Middle East address, called for Palestinian self-government in the territories "in association with Jordan" and Arab-Israeli negotiations "involving an exchange of territory for peace."

Israel would have to withdraw from the West Bank and Gaza Strip in return for Arab recognition of the Jewish state and guarantees for its security, Reagan said. Shamir has always steadfastly opposed any moves that would lead to such an Israeli withdrawal from the territories, which he refers to by their biblical names, Judea and Samaria.

Israel radio further reported that Olmert has told Hill that Shamir is opposed to any automatic "linkage," or "interlocking mechanism" as Shultz is calling it, between talks on interim arrangements for the occupied territories and subsequent negotiations on their final status.

Both Olmert and another close Shamir associate, Dan Meridor, who was here last week, have argued with Shultz that if Jordan and the Palestinians know in advance that the final status negotiations will begin on a specific date, they will never take the initial limited autonomy talks seriously.

Shamir wants to stick to the limited autonomy plan envisaged by the 1978 Camp David accords, which called for a five-year trial period for Palestinian self-rule in the occupied territories before any decision is reached on their final status. Talks on this plan between Israel and Egypt broke down after 1981, and both Egypt and Jordan have opposed their restitution.