The State Department yesterday refused to give Rhodesian Prime Minister Ian Smith a visa to enter the United States.The decision blocks a visit by 15 Rhodesian officials, including the three leaders who serve with Smith on the ruling Executive Council of Rhodesia's biracial transition government.

Sen. S. I. Hayakawa (R-Calif.) and 26 other senators had invited the Rhodesians here to make their case for American support of the transition government's struggle against black nationalist guerrillas.

Although the visit was potentially embarrassing to the administration because of its opposition to Smith's plan for transferring power to blacks, President Carter had indicated that entry visas probably would be approved.

State Department spokesman Ken Brown said yesterday, however, that the visas could not be granted because Smith's government in Rhodesia is illegal. The United Nations has barred the issuance of a visa to anyone holding a Rhodesian passport, and the United States has subscribed to these sanctions.

Among those invited by the senators were the Rev. Ndabaningi Sithole, Bishop Abel Muzorewa and Chief Jeremiah Chirau, the black members of the Executive Council.

The Carter administration has been concerned that a visit by the group could cause increased criticism within United States of U. S. policy toward Rhodesia. The administration has opposed Smith's "internal settlement" that excludes the guerrilla groups fighting in the county's six-year-old war.

Instead, it has called for a settlement involving all groups, a move that Smith says would push an independent Rhodesia in the direction of the Soviet Union , Cuba and China, which support the guerrilla groups.

Hayakawa could not be reached for comment yesterday, but he had said earlier that denial of the Rhodesians' visa request would violate Carter's human rights policies.

Rhodesian officials reported earlier yesterday that Britian had refused even to allow Smith to change planes in Britian should he travel to the United states.

Although Brown said the state Department's review was limited to Smith and other long-time members of the Rhodesian government, he said none of the invited group will "be able to come in time for Sen. Hayakawa's invitation."

The delay was clearly based on the U.N. standards, he said, adding: "We're talking about people who have associated with an illegal regime."

Hayakawa and the other senators had invited the Rhodesians to Washington and New York, and the American Security Council, a private conservative organization, he had made arrangements for the Rhodesians to attend a series of meetings.

"We are disappointed, but more than that, we are outraged at the hypocrisy of the State Department's action," said American Security Council spokesman Philip Clarke.