President Reagan's effort to cut the size of the government has foundered under an avalanche of new mail, a multitude of new weapons systems, millions of aging veterans and the need to collect more taxes and shut out illegal drugs.

Federal employment has gone up 7 percent in the last decade, despite Reagan's efforts, according to the Congressional Budget Office (CBO).

Underlying the total is a historic swing in priorities, with defense employment going up and the number of workers managing the nation's social programs decreasing.

Today, virtually half of all the civil servants in the United States work for the Defense Department, according to a separate study of federal nonpostal employment in the Reagan administration conducted by the staff of the House civil service subcommittee.

And they, in turn, supervise and oversee some 3 million workers in private firms who are engaged, directly or indirectly, in defense-related work, according to internal Defense Department estimates.

"One of the most graphic examples" of government growth, said Sen. David H. Pryor (D-Ark.), chairman of the Senate subcommittee that oversees the civil service, is that the "equivalent of the entire federal work force" is working on defense-related contracts.

"We need to make sure growth is channeled into areas where the need is greatest," he said.

The CBO report suggests that the civil service "may be entering a new era in which recruiting and keeping qualified workers will become increasingly difficult."

The report said that federal retirement changes have lessened the financial incentives for staying with the government, widespread criticism may have lessened the prestige of public service, and federal personnel changes and limited pay increases may have contributed to the phenomena.

The report noted that retirement rates are rising, especially among senior civil servants.

Leonard Weiss, staff director of the Senate Governmental Affairs Committee, said the report "suggests that the more experienced higher-level workers are leaving the government when they can." He expressed concern that this might lead to a reduction in the quality of services provided the public.

The report found that the nature of federal work has changed greatly during the past 10 years. Employees are increasingly white-collar workers engaged in professional and administrative tasks. The number of the archetypical federal employee -- clerks -- actually decreased.

Federal employees are more educated, reflecting their altered duties. About 31 percent have at least a bachelor's degree, up from a quarter of the work force 10 years ago.

Many of the professional occupations experiencing the largest increases -- for example, engineers -- are concentrated in the Defense Department, where Reagan's buildup has occurred.

The growing reliance on contracting out work to the private sector has increased the need for contract administrators -- the number has gone up by 60 percent during the past decade.

Federal employees work in more than 900 occupations and make up about 3 percent of all U.S. workers. Only about 11 percent of them work in the Washington area.

The growth of the federal work force over the last decade is considerably less than the growth of the nation's population, so that the number of Americans served by each federal worker went up from 76 in 1977 to 79 last year.

Many of the new government workers are employed part-time or are temporary employees. These workers, who cost the government less than full-time workers because they are not covered by retirement and other benefits, grew by 9 percent during the past decade.

About 76,000 of the government's new civil servants over the decade work for the Pentagon. The Veterans Administration increased its employment by 20,000 to "cope with the rise in demand for medical and other services that has accompanied the aging of veterans of World War II and the Korean war," according to the CBO report.

Justice Department employment grew by 13,000, or one-fourth, to expand efforts to control U.S. borders and to investigate and prosecute criminal cases involving drugs and white-collar crime, the report said.

Nearly all of the 15 percent increase in Treasury Department employment -- 19,000 -- occurred in the Internal Revenue Service, the report showed.

Chester A. Newland, former director of the Federal Executive Institute in Charlottesville, Va., that trains top federal managers, said that "IRS had to have an increase because the staff cuts had become so counterproductive they wound up actually losing money."

By contrast, the work force of the State Department has declined 16 percent, the Interior Department has shrunk 14 percent; the Agriculture Department 17 percent; the Commerce Department 11 percent, Health and Human Services 17 percent, Housing and Urban Development 28 percent, Transportation 18 percent, and the General Services Administration 40 percent.

The administration, which entered office in 1981 and achieved dramatic cuts initially in the size of the government, coupled that effort with a drive to increase the productivity of those who remained. Although productivity statistics reveal nothing, in the words of the report, "about the quality of government services," they nevertheless show "a steady improvement in federal performance over time."

According to the Bureau of Labor Statistics, the productivity of the federal worker went up at an average annual rate of 1.4 percent in the past decade, a rate that exceeds by about .6 percentage point the rate experienced by the private sector for the same period. Productivity statistics measure the product or service produced per worker.

Although the CBO report, entitled "Federal Civilian Employment," points up the difficulties and inadequacies of productivity statistics, it suggested that some productivity savings are undoubtedly real, and may have allowed the government "to get by with 270,000 fewer civilian workers than might otherwise have been needed."