Sen. Jesse Helms (R-N.C.) charged the State Department yesterday with attempting to "muddy the waters" by accusing his staff of meddling in the Zimbabwe-Rhodesia constitutional conference in London.

Speaking on the Senate floor and later in a telephone interview, Helms said he had been assured by senior British officials that Lord Carrington, the British foreign secretary, had not formally complained to Secretary of State Cyrus R. Vance about a meeting between two of Helms' aides and former Rhodesian prime minister Ian Smith.

A British Foreign Office statement issued in London also categorically denied reports in The Washington Post and The New York Times yesterday that a formal complaint had been lodged with the Carter administration, the Associated Press reported.

But State Department officials strongly suggested that their British colleagues were practicing diplomatic understatement to avoid an open fight with the conservative Republican senator, who has led the fight to have U.S. economic sanctions against the Salisbury government lifted.

Department spokesman Hodding Carter acknowledged to reporters that Vance "did not receive a formal complaint" about the London efforts of Helms staffers John Carbaugh and James P. Lucier. "However, we were aware of the British concern" over reports that Carbaugh and Lucier had urged Smith to hold out in anticipation of U.S. action to end sanctions, Carter said.

A senior State Department official reiterated that British officials had expressed to Vance their "extreme unhappiness" with the activities of Carbaugh and Lucier, and that Vance had been responding to that informal complaint when he telephoned Senate Foreign Relations Committee Chairman Frank Church and Sen. Jacob K. Javits (R-N.Y.) Wednesday.

Church and Javits in turn met with Helms and passed on Vance's Helms said yesterday that he has yet to hear directly from Vance about the conversation with Lord Carrington.

Helms said he called the British foreign secretary yesterday and received return calls from Richard Luce, Undersecretary for foreign affairs, and Roderick Lyne, Private secretary to Lord Carrington, and that both said no protest had been made to Washington.

"This tempest in a teapot is a complete contrivance of the State Department, which trying to muddy the waters and affect the conference," complaint. Helms said yesterday that he has yet to hear directly from Vance about

In a related development, Senate and House Armed Service Committee conferees began discussion of the Defense Department authorization bill. The Senate version contains an amendment that would end the sanctions, enacted after Smith's government declared unilateral independence from British colonial rule in 1965.