Contortions in the State Department to delay a visa for Rhodesian Prime Minister Ian Smith and Executive Coucil member Ndabaningi Sithole symbolized the bankruptcy of Carter administration Rhodesia policy, a failure that has created a vacuum now being filled by the Senate.

Implicit bankruptcy was declared by the Senate more than three months ago when a resolution to remove economic sanctions from Rhodesia barely failed; 42 to 48. That was followed Sept. 14 by a letter to Smith, signed by 27 senators, inviting him and Sithole to Washington. Moderates such as Republican Sens. John Heinz (Pa.) and Bob Packwood (Ore.) and Democratic Sens. Jennings Randolph (W. Va.) and Ernest Hollings (S.C.) were among the signers.

But even such clear warnings from senators determined to arrest the move toward all-out racial war failed to awaken the makers of African policy in the State Department Trapped in a policy that in effect gives veto power over the United States to feuding black states bordering Rhodesia and to feuding guerrilla forces armed and trained by Soviet and Cuban officers, State's African specialists shied from making any gesture to the outlawed Smith.

So when Smith's request for a visa arrived, the State Deaprtment blocked it. At work was the same detachment from reality that has dogged the administration's Rhodesian policy ever since former secretary of state Henry Kissinger's basic plan for ending white domination of black Rhodesia was adopted by Smith last March.

The pretext for the department's preliminary decision Sept. 30 denying the visa was the U.N. resolution imposing sanctions against the onetime British colony. Smith being a government official in an outlawed nation, his passport has no international standing, but the United States can waive that U.N. ban anytime it wants.

For example, both Sithole and Bishop Abel Muzorewa, another member of Rhodesia's Executive Council, got U.S. visas for previous visits here despite U.N. sanctions. The State Department was singling out Smith for special treatment.

But the true hypocrisy of the department's preliminary decision to bar Smith on spurious legalistic grounds is exposed by the fact that Zambia, one of the "front-line" black states bordering Rhodesia, itself waived the U.N. ban and admitted Smith just two months ago.

Smith was invited by Zambian President Kenneth Kaunda for secret negotiations with Joshua Nkomo. The purpose: to find common ground between the two so that Nkomo, a principal leader of guerrilla forces now attacking Rhodesia, could be brought into the Rhodesian government. U.N. sanctions counted for nothing against Zambia's desperate need for a solution to racial war in southern Africa and economic disaster for Zambia and other frontline states.

The political rationale behind the invitation for Smith to come here and talk to senators and other American leaders was grounded on precisely the same hopes that motivated Kaunda: press Smith to find some formula to entice Nkomo into a "share of power" in the Rhodesian government.

The leading Senate player in this game is conservative Sen. Jesse Helms (R-N.C.), who is now working through private channels to soften both Smith and Nkomo in the hope of continuing the August contacts started in Zambia - a meeting that lasted several hours and achieved limited objectives.

But State Department specialists shrank from exposing the shrewd and wily Smith to the U.S. public or risking the political anger of Third World activists in the United Nations and blacks at home. Despite Helms's pleading, Smith's request for a visa languished.

Helms then served notice that he would enter formal objections, under the rule of senatorial courtesy, to the State Department's entire list of foreign-service promotions, and hold up three ambassadorial confirmations. There were other well-founded threats.

Helms, however, is small potatoes in Jimmy Carter's state Department. What broke the visa barricade was not Helms or his Senate colleagues but a compelling editorial in the Washington Post on Oct. 4. By no stretch could The Post be charged with harboring bias toward Smith. Accusing the State Department of playing a "shabby game," the Post asked: "Must the United States be 'purer' than Zambia?" Within hours of that Oct. 4 editorial, the State Department granted the visa, making a mockery of its sanctimonious pretexts for delay.

Having breached the visa barricade, the Senate intends to play out its activist role and fill the policy vacuum that has been spreading since last March, when Smith launched his "internal solution" pointing toward black majority rule. As perceived by senators, the United States for too long has bartered away its prestige and power in the Rhodesian tragedy in a vain search for a solution satisfying black and other interests, many of which are clearly irreconcilable.