Congress may thwart a controversial North Atlantic Treaty Organization plan to deploy new U.S. long-range theater missiles in Western Europe by refusing to approve enough money this year to begin production of the nuclear warhead for the missile.

Members of both houses have become critical of the program because of its sharply escalating costs, the problems of pushing its development to make its deployment date in December, 1983, and growing sentiment in Europe against acceptance of the weapons.

Last July, the House cut $41 million from funding for the ground-launched cruise missile warhead after the Appropriations Committee reported "additional tests are warranted" for the nuclear device and the "planned production rate is unrealistic. . . . "

Although the Senate is expected today to approve full funding of the Reagan administration's plan for the cruise missile's nuclear warhead, congressional sources said it was doubtful there will be any reconciliation with the House before funds run out Nov. 20.

Thereafter, sources said, the entire Department of Energy nuclear weapons building program may have to proceed under a continuing resolution that would provide funds at the lowest level approved by either congressional body.

In the case of the cruise missile, that would be the House figure, which cut some $15 million for special machine tools and $26 million to start making parts for the warhead.

Some key Reagan administration officials believe that if the United States can't start stationing the Pershing II and cruise missiles in Europe by 1983, the whole plan might fall apart. Thus the adminstration can be expected to make a strong pitch to have these funds restored.

Several other nuclear weapons programs directed primarily at European deployments also stand to be affected.

The House cut $35 million that was to initiate production of a new nuclear artillery shell for the 155mm gun. This program was part of a 1977 NATO buildup, pushed by the United States, to increase and modernize nuclear artillery in Europe.

The House Appropriations Committee said that given the decision to go ahead with the new, eight-inch neutron artillery shell, moving ahead with a new 155mm shell was "premature" and "additional analysis of the requirement for this capability is warranted."

Yesterday, the Senate restored these funds, along with $1.2 million to improve the electrical safety system on the old Nike Hercules nuclear warhead.

The House cut that item, saying, "This weapons system is approaching obsolescence" and plans should be made "to remove it from the inventory as rapidly as possible."

Last April, NATO's nuclear planning group approved its own plan to retire the Nike Hercules nuclear warheads over the next five years, but refrained from announcing it publicly. There has been no public explanation why DOE wants to upgrade the safety of these warheads over the next few years.

The latest cost for the overall ground-launched cruise missile program, according to recently released congressional figures, is $2.4 billion for 560 missiles. The expected price tag four years ago, according to the Air Force, was $1.4 billion.