A new Census Bureau study shows that, on average, the elderly are financially better off than the population in general. That fits right in with one of the strongest lifestyle trends in the country: the drive for ever-earlier retirement from the world of work.

An increasing number of middle-aged workers are quitting their jobs to live on their pensions, savings and Social Security. The public question is whether the nation can afford it.

Back in 1948, 90 percent of the men aged 55 to 64 were in the labor force. That number has steadily declined until, in 1980, only 72 percent of men in that age group were still at work. uring the high inflation of the late 1970s, it was thought that more older workers would stay on the job, because of the fear that their money wouldn't last. But so much income-protection now exists for older people that inflation hardly seems to matter. Savings accounts are effectively indexed to the inflation rate (through bank certificates and money-market mutual funds), and Social Security payments rise with the cost of living.

Earlier retirements continue. For the first seven months of 1983, only 69 percent of the men aged 55 to 64 were in the labor force. Among older workers the trend is even more pronounced. In 1948, 47 percent of men 65 and up were still at work. This year, that number is down to 17 percent.

Early retirement has been a blessing to many people. But as growing numbers of the middle-aged leave the work force, labor experts increasingly worry about the cost. Social Security has been in trouble. Corporations are easing away from fixed-lifetime-payment retirement plans. Government spending has been soaring, on such income-support programs as pensions, Medicaid, veterans benefits and housing assistance.

In the larger view, fewer active workers mean fewer taxpayers, and a lower national income, than would otherwise be the case. People in retirement don't produce goods and services, so there's less growth in national output. But they do consume goods and services--creating the classic situation of more people chasing fewer goods.

That combination is inflationary. A 1981 study by Data Resources Inc. figured that if employment continued to decline among older people, it would add 0.2 percentage points a year to the inflation rate.

Students of the labor force say that early retirement incentives are built right into remuneration systems:

In private pensions. Companies make it positively attractive for older workers to move out, so younger workers can move up. Fifty-five percent of the workers sampled by an Urban Institute study had pension plans that could pay full benefits at some age earlier than 65; 24 percent had a normal retirement age of 60 or younger; 11 percent could retire with full benefits, at any age, if they had 30 years of service.

Furthermore, the earlier a worker leaves, the higher his lifetime pension payments are likely to be--another gentle hint to leave. "We found that incentives in private pension plans were the single most important factor in early retirement," James Storey, director of the study, told my associate, Virginia Wilson.

In the past couple of years, companies have been making it especially attractive for middle-aged workers to depart. Early retirements cut the payroll and make room for younger people to move up.

In Social Security. Since 1961, when all workers were first allowed to get social security benefits as early as age 62, younger-age retirements are up dramatically. Twenty years ago, 18 percent of new social security retirees were 62. By 1980, their number had risen to 37 percent. The Social Security reform bill passed last year contains modest incentives to stay in the work force longer, but few observers expect them to make much of an impact. The incentives for quitting a job are simply too great. portion of the early retirees leave involuntarily. Some are effectively forced out of their jobs; some want part-time jobs. Some "retire" simply because they can't find work. Some are in poor health. A 1982 study by the Social Security Administration found that age-62 retirees are somewhat more likely than others to have health problems, but the difference is small. Finances, more than health, dictate the average retirement choice.

Earlier retirements are likely to continue, as older workers are increasingly jostled by the high ambitions of the baby-boom generation. Their leisure--whether voluntary or enforced--will be a depressant on the economy for many years.