The Senate deadlocked anew yesterday over Democratic efforts to constrain the Reagan administration's tanker-escort operation in the Persian Gulf, prompting an angry charge that the Senate was "wandering around in a fog of . . . parliamentary maneuvers" to avoid complying with the law.

The latest impasse occurred as Democrats used a procedural ploy to attempt to force a test vote on their proposed limitations on the escort operations. A countermaneuver by Republicans nullified the vote's significance.

The result was a 99-to-1 vote against killing the measure, which meant only that the impasse continued.

Later in the day, Majority Leader Robert C. Byrd (D-W.Va.), frustrated over failure of an earlier effort to negotiate a bipartisan resolution of the dispute, filed a cloture petition to force a vote Friday on shutting off debate on the gulf issue. Minority Leader Robert J. Dole (R-Kan.) then filed another cloture petition that, if approved, would have the effect of eliminating consideration of the gulf issue as part of the pending defense authorization bill for this fiscal year.

Lashing out at both parties, Sen. Lowell P. Weicker Jr. (R-Conn.) accused Congress and the White House of flouting the law by refusing to invoke the 1973 War Powers Resolution, which would ultimately require congressional approval for continuation of the tanker-escort operation.

So long as government ignores its own laws, "we're no longer a government of laws, we're an anarchy," Weicker said. "I don't think that's an example the Senate wants to set."

Instead of "wandering around in a fog of constitutional and parliamentary maneuvers," he said, the Senate should be voting directly on war-powers compliance as well as other issues that have become enmeshed in debate over the defense bill.

At issue yesterday was a Democratic proposal that sidesteps the War Powers Resolution by invoking many of its provisions in a separate bill to require congressional approval for continuation of the tanker-escort operation after 90 days of enactment of the measure.

Dole characterized the Democratic proposal as a "son of War Powers" under which critics of the administration policies could "take potshots at the president from behind a legal facade" that camouflages their attacks.

The critics are saying they "don't want to shoot down the policy, just fire at it every day or so on the Senate floor," Dole added.

Byrd contended that the reflagging and escorting of Kuwaiti tankers was a "hastily conceived" policy that has resulted in an "open-ended and very expensive operation" and said Congress was justified in demanding some control over its duration.

In yesterday's maneuvering, Byrd attempted to circumvent GOP delaying tactics and force a vote on the issue by moving to table, and thus kill, his own proposal. Under this strategy, supporters of the proposal would vote against the tabling motion, demonstrating enough votes for passage of the measure and putting pressure on Republicans to agree to a vote.

But Dole foiled the strategy by calling on Republicans to join Byrd in opposing the tabling motion. All Republicans wound up voting against it. Its only supporter was Sen. David L. Boren (D-Okla.), who opposes war-powers limits at this time in the gulf.