Prime Minister Yitzhak Rabin retained the leadership of Israel's Labor Pary early today by turning back the challenge of his defense minister, Shimon Peres, in a breathtakingly close party convention vote.

The final vote was 1,445 for Rabin to 1,404 for Peres, with 16 abstentions.

The vote, announced shortly after midnight, ended months of suspense and rivalry between the two men for the leadership of a party that has recently been racked by financial scandals and plagued by the country's growing economic and social woes.

It was the first time that an incumbent prime minister from the Labor Party, which has ruled Israel since independence, has been directly challenged for the leadership. In 1973, after former Prime Minister Golda Meir announced her retirement, Peres made a bid for the party leadership but lost to Rabin.

Today's convention vote means that Rabin is likely to retain his post as prime minister in May's election. Although Labor's fortunes have been declining, the party is still expected by most political observers to win a diminished plurality, with enough seats in Parliament to form a government in coalition with minor parties. But a Labor victory is no longer assured, as it once was.

The vote greatly strengthened Rabin's hand for his visit to Washington early next month.

The Labor Party convention, which opened in Jerusalem Tuesday, moved to Tel Aviv yesterday. While the opening ceremonies in Jerusalem were formal, with most of Israel's leadership wearing suits and ties, the proceeding here were more proletarian, and in line with Israeli dress customs, with most people in shirtsleeves and open collars.

Excitement rose during the final hours while the vote was being counted in the hot and stuffy hall. Delegates tried to sing songs, but to no avail.

Israel has never experienced an open convention of this type - previously candidates were chosen privately in smoke-filed rooms - and the proceedings were televised live. In the last half-hour before the vote was announced it was clear that Rabin had won by the forced smile on Peres' face and joy on the faces of Rabin's supporters.

The two candidates were scheduled to address the convention just before the voting started, but each wanted the other to speak first. Finally Shlomo Hillel, minister of police and president of the party presidium, told the convention that he had decided the matter by flipping a coin.

Rabin, who spoke first, talked about the "moment of truth" that the delegates would have to face when they placed their secret ballot. He stressed country, in which he rose to chief of staff, and his service in the elite Palmach resistance unit in the fight for independence.

This was clearly a jibe at Peres, who - although he has been instrumental in fighting force it is today - never actually served it in the army.

Rabin wrapped himself in the "special fabric" of the Israeli-American relationship, which he said he had helped weave during his five years as ambassador to the United States.

His final point was that the delegates would have consider a vote against him as a vote of no-confidence in the party and what it had achieved.

He said he was not suggesting that prime ministers should serve forever, but that if the party rejected him after only two years and nine months in office it would indicate a lack of faith in the party. If Labor's own delegates showed such a lack of faith, he asked, why should the selectorate vote Labor in May?

Rabin admitted that there had been mistakes, but asked the delegates to let him continue. "I have full confidence that I have done my best," he said.

Rabin was the candidate of the Labor Party's establishment, including Golda Meir, but the closeness of today's vote showed that establishment support no longer carries write the same weight as it once did.

Peres, in his speech, tried to make the point that if the party does not reform itself from within, "Other factors will bring about change from without."

He mentioned the Democratic Movement for Change, ormed by archeologist and former army chief of staff Yigael Yadin, as a reform party.It has been rapidly capitalizing on the general malaise in Israel and the feeling that the old entrenched bureaucracies and politics need shaking up.

The "system" itself has perhaps never before been in such dispute in Israel, and although Yadin's party is not expected to win more Parliament seats than either Labor or the main opposition party, Likud, Yadin is expected to finish a respectable third and may end up as the indispensible coalition partner to anyone who wants to form a government following the general election in May.