President Carter has clouded the future of naval aviation by refusing in his new budget to buy the new plane the Marines want and restricting the Navy to a small carrier, Pentagon sources said yesterday.

Carter, despite pleas by Marine leaders, had knocked out of his fiscal 1980 defense budget the money to continue development of the AV8B Harrier, a fighter-bomber which can take off and land vertically.

The administration's refusal to approve the $200 million sought for the "jump jet" assures a fresh fight in Congress this year over the whole future of such V/STOL aircraft, an abbreviation for vertical and short takeoff and landing.

Marine leaders from Commandant Louis H. Wilson on down have argued that a plane like the advanced Harrier could be based in the field with the troops and provide immediate help in land battles.

Defense Secretary Harold Brown has been skeptical of the new Harrier, maintaining that the Marines have failed to justify its expense. But he kept its development going last year.

Unless the president reverses his decision at the last minute, an unlikely prospect, the whole Marine plan to switch to the advanced Harrier for its future air support will be shot down.

The Marines have already bought 110 of the first generation Harriers the AV8A. But the British-built jump jet has been plagued with 32 crashes. The AV8B was supposed to correct many of the flaws.

Some Pentagon analysts argued that the Marines were going too far into this new type of aircraft and should rely more heavily on Navy planes to support their troops in battle. The Navy buys planes for the Marines, but Marines would fly the advanced Harrier.

Navy officials see irony in the fact that the Carter administration is pulling the Marines back from the V/STOL era while pushing the Navy into it by approving for his fiscal 1980 budget a small aircraft carrier. The small carrier is ideally suited for V/STOL aircraft.

Carter, rejecting the recommendation of Defense Secretary Brown, tentatively has decided to buy a mediumsized carrier of about 60,000 tons instead of the 90,000-ton one Navy leaders requested. Either would be conventionally rather than nuclear powered.

On eargument made during the White House budget deliberations, sources said, was that Congress would be less likely to insist on a Nimitz, nuclear-powered giant in the new budget it offered the the "midi" of 60,000 tons. Congressional backers of a bigger carrier, it was reasoned, might be content to push the 60,000-ton carrier up to a 90,000-ton one like the John F. Kennedy rather than insist on another Nimitz.

Brown favored building a 60,000-ton carrier last year after Carter vetoed the fiscal 1979 defense bill because it authorized another $2 billion Nimitz nuclear carrier he opposed. But Brown changed his mind and reconmended the larger, conventionally powered ship to the president.

Both Navy Secretary W. Graham Claytor and Adm. Thomas Hayward, chief of naval operations, have sent appeals to Carter urging him to approve the larger, 90,000-ton carrier in his new budget. The president had not responded as of last night.