Israeli Prime Minister Menachem Begin, speaking against the background of the fighting in Lebanon, today ignored a massive boycott by anti-Israel delegates to tell the U.N. disarmament conference that "self-defense is a sacred right and duty."

In his address Begin spoke largely about the dangers of nuclear war and made no direct mention of his country's 2-week-old invasion of Lebanon.

But the hostility that Israel's action has provoked among Arab countries and their allies in the Third World and communist bloc was underscored by the fact that 94 of the 157 delegations here were absent during his speech.

Most stayed away. But more than 30 delegations present when the prime minister was introduced stood up and walked out as he began to speak.

They included the representatives of all Arab countries except Egypt, which has made a separate peace with Israel, all the European communist nations except Romania, the majority of African countries and a scattering of radical-leaning regimes from other Third World regions.

As a result, in the same chamber where President Reagan spoke on Thursday before an almost full house, Begin was left with an audience that consisted largely of delegates from the western industrial countries and from Latin America.

Secretary of State Alexander M. Haig Jr., who had a breakfast meeting with Begin before the speech, did not attend because he was preparing for a meeting with Soviet Foreign Minister Andrei F. Gromyko. U.N. Ambassador Jeane J. Kirkpatrick sat in the chief U.S. delegate's chair.

The thrust of Begin's speech was to contend that while "aggressive war should be banned, denounced and renounced" every country has an inherent right to self-defense.