Three former Democratic secretaries of state urged Congress yesterday to cut off secret U.S. aid to anti-government guerrillas in Nicaragua as the House moved toward a vote on the closely contested issue.

Former secretaries of state Edmund S. Muskie, Dean Rusk and Cyrus R. Vance said in a statement that any U.S. support for El Salvador against intervention from the leftist Sandinista regime in Nicaragua "should be overt and not covert." The Democratic-sponsored legislation the House is scheduled to take up today would make the change they advocated.

The three former secretaries of state were joined in their statement by Robert S. McNamara and McGeorge Bundy, who served in the Kennedy and Johnson administrations as secretary of defense and presidential national security affairs adviser, respectively.

"We urge the Congress to enact legislation which will bring an end to the covert operation and which will ensure that American activities in Central America conform to domestic and international law, are conducted openly and are aimed at negotiating a political settlement," their statement said.

House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) told reporters that while today's scheduled vote is likely to be very close a Democratic head count indicates that "we are winning" the battle to cut off the undercover aid from the CIA to the anti-Sandinista guerrillas in Nicaragua.

O'Neill, whose assessment was more positive for the Democrats than those heard from several others in recent days, said 178 Democrats had committed themselves to back the Boland-Zablocki amendment, named for the Democratic chairmen of the House Intelligence and Foreign Affairs committees.

Another 30 to 40 Democrats are "leaning our way," said O'Neill, which would bring the number of supporters close to 218, a majority of the House. The possible addition of 15 to 25 Republicans would produce "a strong vote" for cutting off the covert CIA operation in Nicaragua that might cause the administration to change its policies in Central America, O'Neill said.

Rep. J. Kenneth Robinson (Va.), senior Republican on the Intelligence Committee, said that "it looks like an evenly split House" in which "a good number" of the 267 Democrats would join most Republicans in opposing the cutoff.

Robinson said that, in any case, the vote will be "a total exercise in futility" because the Senate is unlikely to agree to a cutoff of CIA aid to the anti-Sandinista guerrillas and "there is no doubt" President Reagan would veto such a measure if it passed both houses. "I'm confident our vote will be sufficient to sustain a veto," Robinson said.

According to Robinson, the central issue is "the setting of a precedent" in which Congress stops a secret paramilitary operation launched by the executive branch. This was done previously only in the case of CIA operations in Angola, which were halted by congressional action in 1976. Robinson said "we were somewhat differently involved then."

Both Democratic and Republican strategists said the key test for Boland-Zablocki will come on amendments, of which 103 have been filed, rather than on the final vote. Forecasts of votes on amendments are imprecise at best.

In a related action, the House last night approved an amendment to the defense authorization bill prohibiting personnel funded in the measure from being "assigned the mission of overthrowing the government of Nicaragua." The amendment by Rep. Ronald V. Dellums (D-Calif.) was approved by voice vote. The administration maintains that its aid to Nicaraguan insurgents is not for the purpose of overthrowing that government.

While former Democratic secretaries of state were asking Congress to halt the Reagan administration's "secret war" against the Sandinistas in Nicaragua, their Republican counterpart, former secretary of state Henry A. Kissinger, was on Capitol Hill appealing for support for Central American study commission he has agreed to lead.

O'Neill said he emphasized "my opposition to the use of force" in Central America in his meeting with Kissinger.

Senate Majority Leader Howard H. Baker Jr. (R-Tenn.) said he discussed military maneuvers in Central America in his meeting with Kissinger, although such a "short-term" matter is outside the scope of the Kissinger commission's study plan.

Baker said he brought up the subject because it was on the front pages of The Washington Post.