Student enrollment in nearly all categories of education dropped substantially from 1977 to 1978, reflecting a birth-rate decline that began in the 1960s, the Census Bureau reported yesterday.

Total enrollment of all students 3 years old or older in public and private schools was 59.9 million as of October 1978, the bureau reported. It was the first time this decade that total enrollment fell below 60 million.

The bureau said the most noteworthy change since 1977 is the decline in the number of college students - from 11.5 million to 11.1 million. Most of the decline came in the ranks of male students. The number of male and female college students in 1978 was almost identical, but a larger proportion of the females were women over the age of 34 going into adult education programs, possibly reflecting the rise in the number of older women preparing themselves to enter the labor force. There were 1.3 million college students over the age of 34, bout two-thirds of them women.

Although the student population has been declining for several years as the post-World War II baby-boom crop completes the educational process, the number of educational personnel hasn't declined. In fact, it has increased, at least through 1977.

Census Bureau figures show about 6.5 million teachers, administrators and other educational personnel were employed by state and local governments in public schools and colleges in 1977 - about 185,000 more than were employed the previous year.

Officials said reasons for the increase appeared to be a trend toward smaller classes, increases in continuing education evening courses and the addition of specialists for teaching the physically, mentally or eductionally handicapped.

The bureau said the new figures reveal a remarkable change in the college population age 14 to 34 in the past decade. In 1968, only 6 percent of all college students in this age bracket were black. In 1978 the figure was 10 percent (more than 1 million students.)

In 1968 only 39 percent of all college students in this age bracket were women. In 1978 nearly half were women.

The 59.9 million total enrollment figure is 1.4 million less than 1977's enrollment and more than 2 million less than 1975's enrollment. Kindergarten elementary school , high school and college enrollments all dropped from 1977. The only category that showed an increase was nursery schools (1.6 million in 1977 to 1.8 million in 1978). The number of nursery schoolaged children has declined, but an increasing proportion of such children apears to be attending nursery school.