Chairman Hua Kuo-feng, making the first trip by a Chinese Communist party leader to a noncommunist country, arrived in Iran yesterday for a three-day state visit and promptly lashed out at big power "expansionism and hegemonism" - Chinese characterizations of the policies of the Soviet Union.

But Hua's remarks at a lavish banquet given in his honor by Shah Mohammed Reza Pahlavi did not mention the Soviet Union by name and were relatively moderate in tone. Diplomatic sources said Hua did not want to make things more difficult for the shah, who is facing his most serious domestic challenge in a quarter century, by exacerbating his relations with Iran's northern neighbor.

Like China, Iran has a long border with the Soviet Union, and the common interests stemming from that geographical fact were apparent in the banquet toasts.

Hua's trip here following visits to Romania and Yugoslavia is regarded in Iran as part of a Chinese effort to offset Soviet strength in the Indian Ocean area - strength that affects the strategic and oil-rich Persian Gulf. In recent months China has established diplomatic relations with conservative, noncommunist Oman, which occupies a key position on the tip of the Arabian Peninsula at the mouth of the Persian Gulf, and has strengthened its ties with Kuwait.

Most of China's attention, however, has been directed at Iran. Since the two established diplomatic relations in 1971, their ties have been steadily growing closer as each perceives a threat from the Soviet Union. Despite Iran's anticommunist monarchy, the Chinese have been cultivating the shah as an important friend because they view him as a bastion against expanded Soviet influence in the area.

In his toast at the banquet in the glittering Golestan Palace guest residence, the shah referred to the "mutual understand-of China and Iran that states bordering the Indian Ocean and the Persian Gulf should have "sole responsibility" for the stability and security of the two regions.

In his reply, Hua praised Iran for maintaining independent policies and defending its natural resources.

"We categorically oppose and fight the policy of expansionism and hegemonism by the big powers," he said. "The internal affairs of each country should be decided by the people of that country, and the internal affairs of each region should be decided by that region."

The shah is understood to be happy to have strong Chinese support because, as he has repeatedly indicated, he fears he can no longer count on American military help in the event of Soviet aggression. It is not so much a direct invasion that worries the shah as the Soviet Union's ability, in a crisis, to stir up trouble on Iran's borders through its friends Iraq and Afghanistan. The shah is said to believe it useful to have a close friend in China, which is capable of causing similar trouble along its long border with the Soviet Union.

During Hua's visit, the two men are expected to hold at least two and a half hours of talks on world and regional security, Middle East issues, bilateral economic and technical cooperation and their perceptions on their common neighbor.

Hua's visit comes at a time when the Shah is under tremendous internal strain from political and religious foes. A wave of antigovernment riots capping eight months of unrest forced the shah to grant some relatively minor but humiliating concessions to his opponents and to call in a new government to steer the country toward greater political freedom while maintaining a more "Islamic" image.

An example was the front-page newspaper coverage given yesterday to reported government overtures to the Moslem religious leader, Ayatollah Ruhollah Khomeini, living in exile in Iraq, in an effort to ease the present domestic crisis.

The government would not confirm the story, but it created a sensation among Tehran residents, notably in the Bazaar, which has been the scene of much religious unrest and is only a few hundred yards from Hua's Golestan Palace residence.

Police blocked off the street running beside the palace, and contingents of police and some troops were stationed in the area. There were no reports of demonstrations in connection with Hua's visit, which was given little advance publicity or fanfare.

The strain of the recent weeks showed on the Shah's face after he and Empress Farah arrived at the palace for the banquet. Waiting to accompany Hua into the banquet hall, he chatted quietly with the new prime minister, Jaafar Sharif-Emami, who took office Sunday. The Shah's face was grim and he rarely smiled.

When he delivered his toast, the Shah spoke in a low voice that many in his audience had to strain to hear.

Chairman Hua spoke more forcefully and seemed robust and jovial.