Behind the heavily guarded moat of an ancient castle, the foreign ministers of the United States, Egypt and Israel began preliminary talks last night for a new round of meetings starting today aimed at renewing diplomatic progress toward a Middle East peace agreement.

Both the hopes and the aims were modest as Secretary of State Cyrus R. Vance arrived at Leeds Castle, Kent, the 9th Century site picked by British security as the safest place for his meetings with Egypt's Mohammed Kamel and Israel's Moshey Dayan.

The talks were shifted there, and the police and military precautions were sharply increased, after British intelligence received word of a possible terrorist attack planned by Palestinian extremists.

Arriving at a British air base at Manston near the castle southeast of London, Vance said he hoped the two days of Egyptian-Israeli discussions could give a new momentum to the peace process.

After reaching the castle Vance began separate meetings with Kamel and Dayan over the procedures for the three-way talks starting today.

No breakthrough or firm agreements on any major points are expected. About the best that can come out of the sessions, as State Department officials explained it, is an informal dialogue that could start the stalled peace drive on a more productive path through greater mutual understanding and an elaboration of detailed positions.

A specific U.S. objective is an agreement to additional meetings of the two nations in search of more substantive progress within the next one to four weeks.

"This is not a negotiating session," said a State Department briefer who barred use of his name. But he said the castle talks, if successful, should lead to a resumption of fullscale negotiations later this summer.

U.S. officials gave murky and inconclusive answers to questions about the effect on their efforts of last week's meeting in Salzburg, Austria, of Egyptian President Anwar Sadat and Israeli Defense Minister Ezer Weizman.

The United States has received separate reports on those discussions from its ambassadors to Cairo and Tel Aviv.

Vance seemed to downgrade the importance of the Sadat-Weizman meeting by saying as he arrived that the only proposals on the table for discussion at Leeds are Israeli Prime Minister Menachem Begin's limited selfrule plan for the West Bank from last December and the Egyptian counter proposal submitted early this month. Each side has rejected the other's proposals.

A Jerusalem Post report yesterday said Sadat, in his talk with Weizman, suggested that he would agree to permit Israel to retain a security presence and settlements on the West Bank and Gaza Strip in the context of a peace agreement that includes an Israeli withdrawal from those areas after a five-year transition.

The Egyptian government, however, issued a statement denying the report and saying that Israel must "declare clearly and specifically her readiness to evacuate the West Bank and Gaza Strip and terminate the military rule in the two regions . . . before Arabs sit to discuss security arrangements."

The future of the West Bank and Gaza Strip, which have been under Israeli military occupation since the 1967 war, is the most serious sticking point between Egypt and Israel, with Sadat insisting on the principle of eventual withdrawal and Begin refusing to move an inch toward such an agreement.

The West Bank-Gaza question is to be the focus of the talks over the next two days, which may be a major reason for the low U.S. expectations.

Rather than delegations facing each other across a conference table, the United States has arranged "a living room atmosphere" in an effort to generate as much informal contact as possible between the opposing sides. Whether a circle of easy chairs for conversational style will do much to break the Middle East deadlock is not clear, but American officials say any change in setting, procedure and spirit is worth a try.

Vance has let it be known that the will be happy to withdraw from the room if the Egyptian and Israeli foreign ministers can do without him. This will be the first direct talks between Kamel and Dayan since the Jerusalem negotiations broke off in January.

The great worry of the American officials is that without some renewal of momentum, the Sadat initative of last November will die and the Middle East take a much more dangerous turn toward new violence.

"Time is moving on," said a U.S. participant in the talks yesterday. Nothing that it has been eight months since Sadat went to Jerusalem, he said the initiative is "too important" to be permitted to lapse.

The bizarre setting amid the medieval turrets and battlements will lend color as well as security to the proceedings.

The castle, dating from an early wooden structure in 857 A.D., has been used before by the British government for international meetings, but this is believed to be the first time that a kosher cook has been called in and that prayers toward Mecca may be said on its parapets.

Helicopter gunships surveyed the pastoral countryside before the arrival of each party of guests.