The State Department's director of politico-military affairs, Lt. Gen. John T. Chain Jr., yesterday removed most of his retaliatory measures against one of his predecessors, New York Times correspondent Leslie H. Gelb, but stopped short of restoring Gelb's picture to the office wall.

Chain acted after Sen. Charles McC. Mathias Jr. (R-Md.), a senior member of the Senate Foreign Relations Committee, notified the State Department of his intention to summon Chain to explain the basis for steps taken against Gelb.

Chain said he was rescinding a ban he imposed last week on contacts between his aides and Gelb, who headed the bureau from 1977 to 1979. Chain also said he had removed a sign posted in place of Gelb's photograph in a wall display of former bureau directors. The sign said the picture had been removed "for cause" because a Gelb story about nuclear contingency plans involved publication of classified information damaging to U.S. security.

While removing the sign, Chain reiterated his charge yesterday. Times managing editor Seymour Topping yesterday called Chain's continuing accusation "an unwarranted charge against The Times" and "an unfair, unsubstantiated attack on the professional reputation" of Gelb.

The editor said the newspaper and Gelb "went to great lengths, in consultation with State Department officials, to report on sensitive matters in a responsible manner and in fulfillment of the need to keep the American public fully informed." The newspaper on Sunday quoted a senior White House official and Gelb as saying that Gelb had limited his article to contingency plans whose existence had been made public elsewhere, although more sensitive information was in his possession.