In an unusual sign of interagency testiness on arms control proposals the Defense Department has complained that Secretary of State Cyrus R. Vance failed to consult them in advance on a proposed study, officials said yesterday.

At issue is a proposal for a possible international ban on new production of material for nuclear weapons. This would require new weapons to be made from material in present stock-piles.In years past the United States has made similar proposals to the Soviet Union without results.

In a letter to the White House, Vance endorsed the idea for a new study, suggested by Arms Control and Disarmament Agency Director Paul C. Warnke. ACDA asked for the study in a search for proposals to put before the United Nations special sessions on disarmament, which opens May 23 in New York.

When the White House circulated the idea to the Pentagon for reaction last weeK, subordinates of Defense Secretary Harold Brown were described as indignant that Brown had not been consulted in advance.

Pentagon officials said a protest by Brown on Friday to presidential national security adviser Zbigniew Brzezinski toned down the original protest drafted by aides, but complained about both the procedure and the substance.

Pentagon sources yesterday denied, however, that Brown was angry at what they called "a bureaucratic foul-up."

When the United States originally proposed the nuclear cutoff ban, Brown said, it had nuclear superiority over the Soviet Union. Now, Brown said in the letter, a cutoff could preclude the United States from acquiring adequate American-Soviet strategic arms control pact.