Median family income declined by 3.5 percent after adjustment for inflation last year, despite a modest increase in the unadjusted figure, which climbed to $22,388, the Census Bureau reported yesterday.

Thirty-two million Americans, 14 percent of the population, lived below the $9,287 poverty line last year, the bureau said.

That figure was up from 13.2 percent the year before. It was the third year in a row that poverty increased, pushed by a recession that has left 9.5 percent unemployment in its wake.

The poverty ratio has not been as high as 14 percent since 1967.

For blacks, the poverty figure was 34.2 percent in 1981, an increase of nearly two full points above 1980; among Hispanics, it was 26.5 percent, up from 26.2 in 1980.

The Bureau report said that the 1981 median family income of $22,388--the income that half of all families are above and half below--was a nominal increase from the $21,023 recorded for 1980.

But with inflation taken into account, the median income figure for 1981 actually represented a fall in purchasing power by about 3.5 percent. It was the second consecutive year in which the real income of families dropped after adjustment for inflation.

The poverty line is raised each year to reflect price increases. In 1980, it was $8,414 for a family of four. In 1981, it was $9,287.

The bureau said the recession that began in mid-1981, its associated rise in unemployment and the increase in consumer prices all had contributed to the increased poverty, which affected most segments of society. The officially impoverished rose from 29,640,000 to 31,822,000.

One group whose poverty rate was lower in 1981 than 1980 was persons over 65: 15.3 percent compared with 15.9 percent.

The bureau said one reason for this might be that the aged normally aren't working and therefore are less adversely affected as a group by rising unemployment.

Another reason, it said, may be that benefits in the major programs to support the elderly, such as the Social Security program and Supplemental Security Income (welfare), increase automatically each July to keep pace with the Consumer Price Index.

Families headed by a woman with no husband present are a group with one of the highest proverty rates: 34.6 percent in 1981 among all such families, among blacks 55.8 percent and among Hispanics 54.0 percent. Many such families, typically consisting of a mother with several small children, are on welfare.

The increase in the poverty ratio over the past three years reverses the overall pattern of the past two decades. In 1959, the poverty ratio was 22.4 percent for all persons and 55 percent for blacks. It went down steadily in the 1960s, then edged down in 1970s with an occasional upward jolt.

It is unlikely that the new poverty figures reflect the bulk of the Reagan administration's cuts in social programs. Most of those enacted last year didn't go into effect until Sept. 30, 1981 at the earliest and so would affect only the last three months of 1981, the year measured in the report. The information used reflects surveys taken last March in which a large population sample was questioned about its income in 1981.

The Census Bureau, in a recent report that used 1979 as a study year, calculated that if noncash benefits were included, the proportion living in poverty would have been calculated at anywhere from one-fifth to two-fifths lower than under the standard cash-only calculations.