More than one in five American workers was unemployed at some point last year, the highest percentage since the end of World War II, the Labor Department reported yesterday.

The recession's impact fell more heavily on men than women because of the heavy job losses in male-dominated manufacturing, the department said, reporting on an annual survey of joblessness.

The proportion of men with some unemployment was a record 23 percent last year, up from 20 percent in 1981, the previous peak. The proportion of women workers who were unemployed at some time during the year was 20.4 percent last year, compared with 19 percent the year before.

Overall, 22 percent of American workers experienced some unemployment last year. It was 19.5 percent in 1981.

Last year, 26.5 million workers were unemployed for some time, up from 23.4 million in 1981, and 4 million had been without jobs for the entire year, according to the Labor Department. In 1981, 2.9 million people did not work.

Most of the 4 million without jobs all of last year looked for work for only part of the year and then dropped out of the work force. These discouraged workers are not counted by the government as part of the labor force when determining the monthly unemployment rate. That rate dropped during July from 10 percent for civilian workers to 9.5 percent.

The number of people who were employed at any time last year decreased by 500,000 from 116.8 million in 1981. Men accounted for almost all of the employment decline, the Labor Department said.

"The severity of the 1981-82 recession . . . was reflected by both the increase in the number of persons with some employment during 1982 and by a decline in the total with some employment during the year," the Labor Department said. The recession ended last November.

The proportion of men who were employed last year--78.2 percent of the population--was the lowest rate reported since 1948, the Labor Department said. The percent of women employed declined to 57 percent from a level of about 58 percent during the previous three years.

"When you have a recession, it generally hits men harder," said Paul Flaim, a Labor Department economist, because it mostly affects traditionally male-dominated industries such as construction, steelmaking and auto production.

"Most of your factory workers were men," Flaim said. "Women were concentrated in office work" and other jobs in the services area, which wasn't as severely hurt by the recession as factory jobs. "You usually find employment of women increasing very rapidly," Flaim continued. "Women still go into things like nursing that are not affected in a recession," although participation of women in factory work is increasing, Flaim said.

For blacks, 33.4 percent of the labor force last year were unemployed at some time, up from 30.5 percent in 1981. "The job situation in 1982 was particularly serious for black men," of whom 36.5 percent were unemployed at some time last year, the Labor Department said.

The proportion of Hispanics unemployed at any time during last year rose to 27.1 percent from 23.7 percent.

The proportion of blacks who were employed at some time last year was 59 percent, compared with 68 percent of whites who had jobs at some time during the year and 65 percent for Hispanics.