Saudi Arabia has canceled a proposed visit by British Foreign Secretary Francis Pym and hinted at economic retaliation because of the British government's reluctance to receive a high-ranking member of the Palestine Liberation Organization.

The sharp Saudi rebuff presents Britain with what amounts to an ultimatum to change its long-standing policy of restricting contacts with the PLO. At risk could be Britain's influence in the Arab world and--if the Saudi warnings are sincere--export trade worth more than a billion dollars.

The dispute stems from a trip planned here some weeks ago by an Arab League peace delegation led by Morocco's King Hassan and including Farouk Kaddoumi, foreign affairs spokesman of the PLO. The group had already visited France, the Soviet Union, China and the United States.

In the United States, which refuses to deal with the PLO until the group accepts Israel's right to exist, Kaddoumi had not been included in official meetings. British officials said he would be left out of any sessions with Prime Minister Margaret Thatcher or Pym but would be received at lower levels.

The Arabs then angrily called off the visit and rejected compromise proposals put forward by the Foreign Office, which still hopes to salvage the session.

While the Saudis apparently feel unable to shift the American attitude on the PLO, they do believe they can apply pressure on Britain to alter its stance.

"We are trying to get around the difficulty," Pym's deputy, Douglas Hurd, said today in confirming that the Saudis had canceled Pym's visit. He reaffirmed the British view that any top-level contact with the PLO would need to be justified by the PLO's "contribution to the peace process."

Pym had been scheduled to fly to Saudi Arabia this weekend in a hastily scheduled fence-mending trip. But Hurd said that the Saudis insisted that the PLO issue be resolved and the Arab League delegation received first. Pym is still scheduled to go to the United Arab Emirates and Oman.

The extent of Saudi anger was revealed today in a letter published in The Times of London and signed by Bandar Ben Abdullah, assistant minister of the interior. Bandar accused the British of being "foolhardy in humiliating the Arabs."

"The Britain of today is not the one of yesteryear," he wrote. "In terms of political and military might it is no more than an appendage to the United States, so it should have gracefully accepted the Arab League's call, because Britain in terms of influencing events in the area is almost irrelevant."

Bandar said that "the recent humiliation" of the Arab delegation that went to Britain to explain the Arab position regarding the Palestinian problem "should not go unnoticed--or unpunished."

His letter concluded by urging other Arabs "in response to this British insult" to follow "the Saudi way--namely hit the Westerners where it hurts--in their pockets."

Last month when former foreign secretary Lord Carrington visited Saudi Arabia on a business trip, he was also warned of trade retaliation and, according to news accounts, passed the word to Thatcher.

In December, the Saudis awarded a major defense contract to the French, specifically choosing France over Britain, which had also vied for the deal.

The Arab League delegation, which also includes Saudi Foreign Minister Prince Saud Faisal, was set up at the Fez summit last September. Its mission was to explain to the five permanent members of the Security Council the Arabs' eight-point peace plan for the Middle East.

One of the main architects of that plan was Saudi King Fahd, which may explain the depth of Saudi anger at Britain's effort to place conditions on meeting with the delegation. Last summer Hurd became the highest-ranking British official to meet with the PLO, but Thatcher has apparently refused to upgrade contact with the organization any further.

Ironically, Britain's relations with Israel also have been severely strained in recent months by the government's outspoken criticism of the policies of Prime Minister Menachem Begin's government, particularly the invasion of Lebanon last summer. The British then imposed a symbolic arms embargo on Israel, which does not buy weapons here anyway.

Just before Christmas, the British announced they would send a small contingent to join the United States, France and Italy in the multinational peace-keeping force in Lebanon.

In 1980, the Saudis, angry over a British-made film drama depicting the public execution of a Saudi princess for adultery, asked Britain's ambassador to leave. British trade with the Saudis also was hurt. In his letter today, Bandar said the British should have learned that the "Saudis can only be pushed around so far."