Since 1970, as the tidal wave of the baby boom generation rolled past, the number of students enrolled in the nation's elementary schools has plummeted by 20 percent, or 6.5 million, according to a Census Bureau survey released yesterday.

High schools have been feeling the ebb since 1976. They have lost at least 1 million students, half of them in the last year, and the emptying of their classrooms will accelerate.

The survey of school enrollment trends depicts a national student body which, while dwindling in numbers, has become dramatically more diverse in age, race, sex and other characteristics.

Women attending college now out-number men. Since 1975, 1 million more women than expected enrolled in college while the number of men declined by half a million.

"Most of the extra women are older," noted Rosaling Bruno, whose office prepared the survey. The rates of attendance are now about the same for men and women of regular college age. Of the older female students, she said, "Most are employed, and most attend college only part time."

In addition to 10 million students under 34 enrolled in college, there were 1.2 million older students, many working or retired. Women in this group outnumbered men 2 to 1. The number of adult students increased rapidly in the early 1970s but has fluctuated since, and last year showed the first decline for women since the bureau started counting them in 1972.

The decline in the attendance rate of men is partly the result of two "military" factors, the report suggests. One was elimination of a large group of veterans from eligibility for GI Bill education benefits after 1975, Another was the end of the Vietnam war, when large numbers of young men were discharged from military service, causing a change in the population base of men considered eligible for college and on which the rates were computed.

Half the growth in college enrollment over the last decade was in part-time students. Among students under 35 years old, nearly half the growth was in two-year colleges, which attract many part-time students. By 1980, 30 percent of college students under age 34 were attending part time.

"Colleges have become more accessible than they were 10 or 20 years ago" when travel far from home often was required, said Vance Grant of the National Center for Education Statistics, referring to the increase in two-year colleges. "In a way, they've brought college to the students."

In all, from nursery to college, more than 57 million Americans between the ages of 3 and 34 were signed up, for schooling last fall when the survey was taken. That is a drop of 3.6 million in five years.

But 3- and 4-year-olds continue to toddle to class in record numbers, nearly doubling the attendance at nursery schools in 10 years and tripling it since 1965.

This is not attributable solely to an increase in the number of working mothers who need a place to leave youngsters, Bruno said, "because only about half of the children in nursery schools have mothers who work" and only 30 percent of the children attend nursery classes full time.

The increase reflects also a change in attitude toward pre-school education, she said. "Now, there's more emphasis on socialization of the children, as well as on learning to read and write."

The college attendance rate of black students, which rose dramatically during the first half of the decade, has leveled off and blacks are going to college at about the same rate as the rest of the population, the survey notes.

But those rates are based on the eligible pool -- those who graduated from high school. And blacks continue to graduate from high school (and from college) at a lower rate than others.