A widely publicized visit to Washington this week by four Rhodesian blacks backing the white-dominated government of Prime Minister Ian Smith appears to have collapsed after the State Department refused visas for two of them.

Two U.S. Senators and the private, conservative American Security Council protested handling of the visa applications. A spokesman for the council called it "contrary to the principle of freedom of speech."

The Rhodesians were to have addressed the American Legion, Veterans of Foreign Wars, the American Security Council, Senate staff aides and the private Heritage Foundation.

None of these groups acknowledges helping to pay for the Rhodesians' trip, which ended in an apparently fruitless wait for visas at the U.S. consulate in Munich. Kae Lucas of Verlin Associates said she handled the appointments here as Washington representative of writer Robin Moore ("The Green Berets") "who has an unofficial American embassy in Salisbury."

Philip Clarke of the American Security Council charged that the State Department was deliberately excluding the Rhodesians, whom he described as being from an "apparently moderate faction," because their presence could detract from the chances of an Anglo-American plan for replacing Smith with a government elected by universal suffrage in the predominantly black country.

Sens. Harry Bryd Jr. (Ind-Va.) and Clifford Case (R-N.J.) cabled protests to Secretary of State Cyrus Vance.

Assistant Secretary of State for African Affairs Richard M. Moose Jr. said that two visas were denied because the applicants - tribal chiefs who are also senators - are, through their membership in Parliament, representatives of the Smith government. Under U.N. sanctions imposed on the breakaway British colony, all nations are to deny visas to Rhodesian officials.

"Any suggestion that there was any political question, other than the visa policy, involved is a baseless canard," said Moosse, who pointed out that he and Ambassador to the U.N. Andrew Young met with at least one of the would-be visitors when the U.S. officials were in Rhodesia recently.

Moose said that his office granted a waiver for two other black Rhodesians on the assumption that they were not official representatives. A cable from Munich indicated, however, that all were returning to Rhodesia without any actually having received visas.

The applicants include:

Senator-Chief Jeremiah Chirau, leader of the newly formed Zimbabwe United Peoples Organization (ZUPO), described in a South African press release on Rhodesia as the only black nationalist group "prepared to talk to the Rhodesian government without immediate British involvement." He was denied a visa.

Senator-Chief Edgar Musikavanhu, also denied a visa.

Chief Kawisa Ndiweni. He was granted a waiver for the visa but Lucas' information indicates he is in fact a senator.

Waivers also were granted to an interpreter and a white Rhodesian, John Stewart, accompanying the group.

Asked who was paying for the trip, Moore's representatives, Lucas said. "I wish I could answer that. As far as I'm concerned they're private citizens."

Another public relations representative, Milton E. Mitler of Fraser Associates, said his firm was asked by that of Lucas to set up a press conference for the Rhodesians on Wednesday.

The press conference is now expected to be canceled, but the American Security Council plans to go ahead with a Tuesday lunch for the media at the Cosmos Club, leaving empty chairs for the Rhodesians.