Two years after first agreeing with the NATO allies to do so, the United States has begun rapidly reducing its older, short-range, tactical nuclear weapons stockpiled in NATO countries.

The existing stockpile of older nuclear bombs, artillery shells, missile warheads and nuclear land mines, which grew to 7,000 during the 1970s, will drop to 4,550 within a year, according to Defense Department officials.

The reductions are being made under a plan drawn up by Gen. Bernard W. Rogers, supreme commander of North Atlantic Treaty Organization forces, after a disagreement with civilian officials in the Pentagon. The Rogers Plan also calls for introduction in Europe of modernized U.S. nuclear systems.

For example, Rogers' plan calls for lower numbers but new versions of nuclear artillery shells. He told Congress last year that new 8-inch shells produced in 1985 would be deployed to Europe "after consultation with the allies," according to recently released congressional testimony. One congressional source said more than 100 such shells were produced last year.

New 155-mm. nuclear shells will also be sent to Europe, but production will not begin for several years.

Rogers' plan also envisions future modernization of the old, 60-mile-range Lance nuclear missile and development of a totally new, 250-mile-range nuclear weapon. The 30-year-old Nike Hercules nuclear antiaircraft missiles will be replaced by the Patriot conventional air defense system; other conventional weapons are to take over roles now handled by nuclear systems.

Rogers outlined his proposals last year in a closed session of the Senate Armed Services Committee. A declassified version of that congressional testimony was recently published.

He also told the senators that the United States would not send extra, or "reload," Pershing II or ground-launched cruise missiles to Europe. Some critics of the controversial U.S. deployment of these medium-range missiles in Europe have said the American force would be above the 572 approved in 1979 by NATO because launchers for both weapons could be fired with "reload" missiles.

While agreeing that reloading the launchers was "feasible," Rogers said, "there are no plans or preparations to deploy additional missiles."

The decision to eliminate some nuclear weapons based in Europe was originally taken at the 1983 NATO defense ministers' meeting in Montebello, Canada. The ministers called for withdrawing 1,400 of the then-estimated 6,000 U.S. nuclear warheads deployed in Europe. In a tense session at Montebello, Rogers refused to accept the reductions as proposed by a NATO working group headed by Richard N. Perle, assistant secretary of defense, and insisted that his staff decide which weapons to cut.

"Rogers delayed the decision and then came in with a few below" what the Perle group recommended, a Reagan administration official said. He added, "There is no foot-dragging now."

The number of U.S. tactical nuclear weapons in Europe may drop even further in the future, sources said, because of a congressionally mandated limit on the number of new U.S. nuclear artillery shells that can be built. At one point, there were 2,000 U.S. nuclear artillery shells in Europe for 8-inch and 155-mm. guns.

In 1984, however, Congress limited the Pentagon to 925 new artillery shells of both types. Only 600 of them can be sent to Europe, a congressional aide said, because the rest were built by the Reagan administration before 1984 and are neutron 8-inch shells, the controversial weapon that creates a high radiation yield that NATO leaders have refused to accept on their territories.

Some top Army officers believe that within five years, new high-tech conventional weapons with great accuracy and high yields may replace almost all battlefield nuclear systems.

In the past, when the military value of many of the currently deployed weapons had been questioned by Congress, officials in government and some outside experts, NATO military and political leaders balked at reducing the nuclear stockpile without getting similar reductions from the Soviet side.

The only previous cut came in 1979 as part of the decision to send 572 new American medium-range nuclear missiles to Western Europe. At that time the alliance approved a 1,000-warhead reduction.

After a five-year study, however, Rogers apparently has joined those who had challenged the usefulness of some of the weapons. He told the Senate Armed Services Committee recently that his experts told him that "if you fired an atomic demolition munition" -- the nuclear land mine with a yield equal to that of the Hiroshima bomb, which has been deployed in Europe since the mid-1960s -- "you can't use that soil for sooner than one year and maybe as long as five."

"The thought has to cross your mind," Rogers continued, "where are you going to use those weapons even if authority were given?"

The 450 nuclear mines now in Europe are among the weapons to be removed, sources said.