The Pentagon has ordered a new study on the problems in fighting a "little" nuclear war - one confined to a battlefield rather than an all-out nuclear exchange.

Of special interest is whether improved communications would help keep track of tactical nuclear weapons and, if war broke out, whether those communications could survive enemy attack.

Defense Secretary Harold Brown has long been skeptical of whether the superpowers could refrain from going to all-out war once either side fired a nuclear weaponof any size.

"My own judgment is that once one starts to use nuclear weapons, even in a tactical way, it is quite likely that it will escalate," Brown told th House Appropriations subcommittee on defense at hearings this year.

"There is a kind of power train," said Brown in explaining his doubts about confining a nuclear war, that goes into motion "even if both sides do not want it to happen."

"The compression of the time for decision," he continued, "the lack of information that would be available on both sides, the expected great advantage that a military commander might think would come from being the first to get in his blow, all push for rapid escalation.

"On the other hand," Brown told the House, "I do not think it is entirely inevitable. For that reason I think we need a spectrum of different kinds of nuclear weapons and close command and control so that if, for example, Europe is being subjected to a massive conventional attack, a few nuclear weapons can be used with the thought that perhaps - and it is only perhaps - that will be a strong enough signals so that the two sides will stop and try to find some other solution."

"I do not have great confidence in that at all," Brown said, "but I think it is worth preserving the option, providing we remember that it is by no means assured - in fact, in my view - unlikely that things could stop that way."

The Pentagon's Defense Nuclear Agency notified U.S. contractors through a notice placed in the government's publication "Commerce Business Daily" of Nov. 2, that it will finance a study of how to protect and command battlefield nuclear weapons.

The study, according to the agency, will include how present and planned equipment for communicating, commanding and controlling a battle would improve "theater nuclear force effectiveness."

There also will be tests, the agency said, of ideas for the "survivability and security of the theater nuclear forces."

Some Army field commanders in Europe complain that it would take too long to receive permission to fire a tactical nuclear weapon to do any good in a battle.

Commanders also complain that NATO leaders have failed to come up with an adequate plan for evacuating tactical nuclear weapons from Germany if they are in danger of being overrun by Warsaw Pact forces.

The new nuclear agency study is one of several fresh looks at the problems in waging "limited" nuclear war.