The Reagan administration tried yesterday without immediate success to convince visiting Australian Prime Minister Malcolm Fraser that his county should participate in a multinational peace-keeping force in the Sinai Desert.

Australian participation in this force was the major unresolved issue after two hours of talks between President Reagan and Fraser that were described by both sides as exceptionally friendly.

A senior White House official said the United States recognized that the issue was "a sensitive and delicate one" for Australia, which is trying to perserve and expand trade relations with Arab nations, but that he expected that Australia ultimately would contribute to the 2,500-member force.

Participation requires approval by the Australian cabinet, and Fraser had made it clear in advance that he would not make a commitment while in Washington.

However, U.S. officials hope for an Australian decision "as soon as possible," one administration official said. They say they believe that Australian participation will encourage New Zealand and other nations to send troops for the force, which is to police the Sinai after the scheduled withdrawal of Israel next spring.

The force is being organized by the United States in keeping with a commitment made by President Carter at the time of the Egyptian-Israel peace treaty. The United States is expected to provide about half of the force.

Because of the ideological compatibility between Reagan and Fraser and the tradition of friendship between the two nations, there are high expectations here that Fraser will lead Australia into supplying at least a token contingent for the force.

Australia is concerned that Arab nations that oppose the Israeli-Egyptian treaty will refuse to buy its wheat and wool. Some Australians also want to limit any participation in overseas military forces to those organized by the United Nations.

Fraser was warmly welcomed by the president yesterday in a ceremony on the South Lawn that included a 19-gun salute and a colonial fife-and-drum corps that played "Waltzing Matilda."

"America is proud to have such an ally in a world where freedom and democracy are constantly challenged," Reagan said.

Fraser responded in a smiliar vein, saying that the two countries "share a commitment to the values of freedom and of democracy."

"There are so many things that will not be done unless the United States is prepared to do them," Fraser said. "There is so much that only the world's greatest democracy can do."

Later, after a 45-minute private conversation, Fraser praised Reagan's commitment to relieving the effect of U.S. antitrust laws on companies doing business in Australia. The law has involved Australian uranium firms doing business with U.S. companies in lawsuits, and Reagan told Fraser that Attorney General William French Smith would discuss the issue with Australian officials in an attempt to solve the problem.

On his first day back in the White House after a five-day western trip, Reagan also met with some congressmen, including Jack Kemp (R-N.Y.), who emerged from the White House saying that he is optimistic that the House Democratic leadership will deliver a tax-cut bill to the White House by Aug. 1.

Another visitor was Senate Judiciary Committee Chairman Strom Thurmond (R-S.C.), who left the White House with Miss Universe Shawn Weatherly, on one arm, and Miss U.S.A., Kim Seel Brede, on the other. Asked about the administration's views on extension of the Voting Rights Act, Thurmond replied: "We didn't discuss no business, just pulchritude."