Nearly 16 million American families, or about one family in five, incur "catastrophic" out-of-pocket medical costs each year, according to a study funded by the government's National Center for Health Services Research.

Many of these families are headed by elderly or unemployed persons, and about a third have income below the government's official poverty line, the study found.

The study is one of the first to estimate how many Americans have potentially ruinous out-of-pocket medical costs when a severe or long-term illness hits. NCHSR studies are considered to be highly reliable by many health professionals and are often used by government policy-makers.

In the study, catastrophic costs were defined as out-of-pocket expenses -- costs beyond those covered by private or public health insurance or by Medicaid or other government programs -- exceeding 5 percent of the family's gross income.

The study's principal authors, S.E. Berki and Leon Wyszewianski of the University of Michigan School of Public Health, said that people suffering catastrophic costs were of two types: those who had good health insurance but had very large costs beyond their coverage, and families "for whom relatively small out-of-pocket expenses represented a high percentage of their income due to a combination of low income and inadequate or nonexistent health-care coverage."

Concern over catastrophic medical costs, which in some cases reach tens of thousands or even hundreds of thousands of dollars a year, has been mounting. On Feb. 4 President Reagan ordered Secretary of Health and Human Services Otis R. Bowen to conduct a one-year analysis of the best ways to provide catastrophic insurance protection to Americans, using the private insurance industry to the largest extent possible.

The new study of the extent of catastrophic expenses was based on information derived from the NCHSR's 1977 health data base, but Wyszewianski said in a telephone interview that since poverty is now greater than in 1977 and Medicaid benefits in some ways more restricted, the proportion of families with catastrophic costs may be greater today than in 1977.

The study found that 8.1 million families had out-of-pocket expenses of 5 percent to 10 percent of gross income; 4.2 million had expenses of 10 percent to 20 percent, and 3.3 million had expenses over 20 percent.

Of the total 15.6 million families who faced catastrophic expenses, 31.5 percent had incomes below the poverty line. A third of all the families were headed by persons age 65 or over. Only 47 percent of the families in the catastrophic class were headed by persons employed all year.

Families classified by the study as falling into the catastrophic class averaged $1,059 in out-of-pocket medical outlays, on top of about $1,800 paid by insurance or government programs. In 1977, according to the study, the 15.6 million families constituted about a fifth of all U.S. families but had 42 percent of the nation's total health expenditures, including both those covered by insurance and out-of-pocket expenses.