Public school enrollment in Prince George's and Montgomery counties dropped this fall, school officials report, continuing a trend that began a decade ago because of declining birth rates and the increased number of children attending private schools.

But private-school administrators say their enrollment levels are being maintained -- and in some cases increased -- and most have growing waiting lists. Dissatisfaction with the quality of public school education, frustration over school budget constraints and recent school closings have prompted many area parents to transfer their children to private institutions.

Ten years ago, there were 161,800 public school students in Prince George's; this fall there are around 111,800. In Montgomery, enrollment has dropped from 126,300 to 92,500 over the decade, school officials said.

Although 1982 figures on the number of students enrolled in private schools are not yet available, the state Department of Planning estimates that 78 percent of the school-age children in Prince George's and 82 percent in Montgomery attend public school. Last year about 80 percent of school age children went to public schools in Prince George's; the rate in Montgomery has remained about the same for the past two years.

The Catholic Archdiocese of Washington reports that enrollments have risen slightly at Catholic schools in Montgomery and Prince George's in the past year, after a dip in enrollment at the end of the 1970s. In Prince George's, the number of students enrolled increased by 111 students between the fall of 1980 and the beginning of classes in 1981. Montgomery Catholic schools gained 80 students during that same period.

As public school systems close facilities because of declining enrollment, private schools are moving into them: In Montgomery County this year, 11 closed schools have been converted to private school use, while one was taken over in Prince George's, officials said.

When Montgomery County surveyed parents who transferred their children to private schools, the report stated that they wanted "a more individualized environment, smaller classes, and programs that they feel will meet more effectively their children's academic needs."

Berwyn Heights resident Ethel Jacobs said it was "the closings as well as the lack of discipline" that prompted her and her husband, Michael, to switch their children from a public school to the nearby Holy Redeemer Catholic school last fall.

The Jacobs said their unhappiness stemmed from having to move their children from College Park Elementary, when the Prince George's school board closed it two years ago, to Berwyn Heights Elementary. It too was closed, a year later. "We couldn't put their three youngsters through that again," she said. "I feel that they learned more in six months [in the parochial school] than they did in one year [in the public schools]," Jacobs said.

In Montgomery County, Chevy Chase resident Katie Buell said she transferred her son to the third grade at Holy Cross Elementary in Garrett Park this fall "as a direct result of the closing of North Chevy Chase School and moving it to Rosemary Hills."

Buell, who was to have been the PTA vice president at North Chevy Chase this year, said she felt like "a rat leaving a sinking ship" when she moved her son. But she said that she "would rather pay for his education and know that the school was going to stay open." Buell, who has three pre-schoolers at home, said she and her husband probably will send them to private schools as well.

State and Catholic school officials report that parochial schools still enroll most of the private students in both counties. Msgr. Edward Spiers, director of research and planning for the Archdiocese of Washington schools, said parochial school enrollment hit its lowest point in the 1979-80 school year, but is now on an upswing.

At Holton-Arms, a prestigious girls school in Bethesda, the number of applicants has continued to rise, especially for the seventh and ninth grades, according to admissions director Noelle Vitt. Most of the applicants live in the District and Montgomery County, she said.

Applications to the school tend to rise when there are major changes taking place in the public schools, Vitt added. This year a number of new students have come from Silver Spring, where there were several school closings.

St. Andrews, an Episcopal school in Bethesda, has grown from 40 to 300 students since 1978. The school moved from a church basement into the old North Bethesda Junior High in July 1981. Alice Anne Freund, the school's staff director, says the rapid growth was in response to the demand for additional private, coed, religious schools in the area -- schools where "parents feel they have more control over what's going on."

Freund says that St. Andrews, which serves grades seven through 12 at a cost of $4,100 per student, receives more applicants than it can accept from families throughout the metropolitan area.

The limping economy does not seem to be hurting the enrollment at most area private schools either. Ritalou Harris, executive director of the Association of Independent Schools of Greater Washington, said that even when tight dollars force families to withdraw their children from private schools, "there are other families ready to take their place."

Even when tuition goes up, parents "go to great lengths -- sometimes getting a second job -- to keep their children in the school," said the registrar, Caryn Kozel, at Riverdale Baptist School in Upper Marlboro. That school enrolls 1,075 students in kindergarten through 12th grade at a cost of $1,380 annually for each student.

There are families who do see improvements in the public school system, however. Diana McCuster, president of the county's Council of PTAs, says Prince George's schools now include a wider variety of course offerings in middle and high schools and two science and technology schools (Oxon Hill and Roosevelt), where students must pass an exam before entering.

McCuster believes these improvements are bringing students back into the public schools. "We see an increase (in enrollment) in middle school and high school," she said.