Critical decisions on defense spending and policy that could significantly affect Britain's role in NATO face Prime Minister Margaret Thatcher's new defense secretary, John Nott, who brings with him a reputation as a budget-cutter.

In a growing debate here over defense priorities, some experts, politicians and commentators are questioning whether Britain can maintain its many expensive military commitments while its economic base continues to shrink. They say Britain eventually may be forced to choose among its independent nuclear deterrent, its naval and air defense of the North Atlantic or its contribution of 50,000 troops to NATO forces in West Germany.

Over the strong objections of Nott's predecessor as defense secretary, Francis Pym, Thatcher and her Cabinet already have decided that defense must share in the government spending cuts dictated by her monetarist strategy for restructuring the economy.

Nott must decide how to trim planned defense spending for the coming fiscal year by as much as 6 percent, including about $500 million in cuts ordered by the Cabinet and possibly another $1 billion to compensate for overspending allowed by Pym.

Military chiefs and defense contractors fear that Nott might cancel or delay any of a number of expensive advanced weapons systems, including a new antiship missile, an air defense missile, an antisubmarine helicopter and new fighter planes, warships and tanks sought for the 1990s.

Sources close to Thatcher say they have tried to dispel "any concern abroad" that Pym's replacement by Nott, a staunch monetarist, means a reduced commitment to defense spending.

"She does not want to reduce defense spending," said one. "The only question in her mind is how far you can pursue expansionist defense budgets regardless of your economic health."

Thatcher confirmed recently that Britain would increase its defense spending by 2.5 percent above inflation during the current fiscal year, compared to the NATO target of 3 percent and Thatcher's originally budgeted increase of 3.5 percent. She would not specify the next fiscal year's increase, but some government sources doubt that it will come even as close to the NATO target as this year's.

Thatcher has asked Nott, who had been trade secretary, to help ease the financial squeeze by increasing the export of defense technology and hardware.

Treasury officials also say an efficient administrator can still trim significant waste from defense spending without affecting military capability, despite the savings Pym already achieved on fuel consumption and training exercises. Some sources compare Nott to President-elect Reagan's choice for defense secretary, Caspar Weinberger, who also brings a budget-cutting reputation to the job.

Nott's room to maneuver is limited by the 55 percent pay raise Thatcher has given the military since becoming prime minister and decisions to buy expensive new aircraft carriers, planes and submarine-launched nuclear missiles.

This leaves vulnerable smaller scale advanced weapons packages such as the antiship and air defense missiles or longer range procurement plans to which the government is not yet contractually committed and could delay.

Julian Critchley, a Conservative member of Parliament specializing in defense issues, suggested that Nott could bring "a fresh mind" to reevaluating British defense commitments. "He's just the man to come into what for him will be a new field and decide it is more sensible to have a fundamental review rather than starve all arms of the military for the next three or four years," Critchley said, expressing a view gaining support. "The incontrovertible fact is that we do live above our station. Something must go."

Britons with this view see the choice as being among Trident, which will cost at least $12 billion to replace Britain's aging Polaris submarine nuclear deterrent; defense responsibilities in the North Atlantic, which require constant and expensive technological advances in ships, planes and weapons to match the Soviets, or the 50,000 troops in West Germany, whose cost has soared as West Germany phases out financial support.

Pym recently sought to quash speculation that Britain might consider pulling out the troops. "Let me make it clear that Britain's treaty commitment of major land and air forces in [West Germany] is central to our whole alliance contribution," he said.

Pym acknowledged, however, that he has launched an initiative within NATO for a study of whether the allies could be more cost-effective in their sharing of commitments and spending at a time of economic retrenchment.

Government sources said they expected this initiative to be pursued in NATO by Nott and by Thatcher's foreign secretary, Lord Carrington, after the Reagan administration settles in and developments in Poland occupy less of the alliance's attention.