The problems of declining enrollment and declining achievement plus a touch of teacher militancy face Washington-area school systems this week as about 520,000 students return for fall classes.

Because of falling enrollment, 13 schools throughout the area will not reopen and teaching staffs have been trimmed. Even so, dozens of schools will have surplus seats and classrooms, and average class size in most school systems is expected to drop slightly.

To try to raise academic achievement, 9th and 10th graders in Maryland and Virginia will be required to take new statewide competency tests that they eventually will have to pass before graduating from high school. Similar tests have been promised for Washington schools, but have not been developed.

After two years of relative quiet teacher militancy has sharpened in Arlington and Washington. Unions in both school systems have vowed to "work to the rule," limiting extracurricular activities and after-school work, because of disputes over salaries and contracts.

Students are scheduled to start classes Tuesday in all of the suburban public school systems. In Washington, the fall term begins Thursday.

The enrollment decline in the Washington area is part of a nationwide trend, caused by a falling birth-rate. But differences in the amount of housing being built, the age of neighborhoods, migration patterns and school desegregation policies have caused considerable variations among different school systems.

Generally, the declines have been sharpest in the older parts of the metropolitan area - Washington, Arlington and Alexandria, and sections of the surrounding counties tht lie inside the Beltway.

In Prince George's County, which has had a major court-ordered busing program since 1973, the number of whites in the school system has dropped by about a third in six years. Meanwhile, the number of black students, many from Washington, has increased.

Alexandria, the only other local school system with large-scale busing, also has lost large numbers of whites. But last year the number of blacks started to fall too, and the decline in whites slowed substantially. Officials believe the racial composition of the Alexandria school system may have stabilized with about equal numbers of black and white students.

Because of the overall decline in students, the Alexandria school board has closed two elementary schools during the past two years. Both of them - Lee and Cora Kelly - are in predominantly black areas and the choices provoked heated protests by blacks.

Last month Alexandria NAACP filed a lawsuit charging discrimination. Officials said the suit will not caused any changes in school plans this.

Even where racial issues are absent, school closings have caused considerable controversy. In Washington last winter a proposal by Superintendent Vincent Reed to close 23 schools met opposition at a series of public hearings. In June the school board voted to close nine old buildings, six of which will be empty this fall.

During the past three years, Prince George's County has closed 10 public schools and Montgomery has closed 21. Two of Montgomery's closed buildings are being used by private schools; two as public school centers for handicapped children, and one is being turned into a senior citizens center. Most of the others are now administrative offices for the school system or other parts of the country government.

In most school systems the decline in student achievenemt, measured by standardized tests, has been gradual over the past 10 years and now appears to be leveling out.

The most recent test results indicate that three of the area's school systems - Fairfax, Montgomery and Arlington - rank high by national standards. Two - Alexandria and Prince George's - are slightly below national norms. Average scores for District of Columbia schools are low.

During the past year both the Maryland and Virginia state boards of education have voted to require students to pass minimum competency tests before they can graduate from high school and to begin phasing in the tests this fall.

In Maryland, students in the class of 1982 will be required to pass a functional reading test. It will be given all 9th graders in October. Those who fail will be offered remedial work and will be allowed to take the test each year until they pass, but will not be able to graduate unless they do.

In Virginia, the new state test requirement covers both reading and mathematics. It will be phased in over the next three years, starting with those who are 10th graders this fall.

Tests in both states are at a fairly easy level, measuring skills normally taught in elementary and junior high schools, such as reading directions and forms or figuring discounts on merchandise. Officials in both states are uncertain what proportion of students will fail and how effective remedial programs will be.

The Washington school board passed a resolution in July 1977 declaring that minimum standards of achievement, enforced by citywide tests, would be required for graduation from high school. But so far no standards have been established and no tests have been developed. Reed said he is uncertain when they will be ready.

In addition to the new tests, local school systems are using new programs to raise achievement, all of which turn away from the emphasis on flexibility and choice that marked the education reforms of a half decade ago.

The Arlington school board is opening a "traditional" elementary school that has attracted 70 more students than it can accommodate. Fairfax County is requiring all junior and senior high students to write at least two research papers a year.

Montgomery County is expanding programs in mathematics and language arts. The programs use computers to help with the extensive testing and recordkeeping required.

In Washington the school system is moving ahead with Reed's "competency-based curriculum," which also emphasizes step-by-step instruction and frequent tests. It will be used to teach reading, writing and mathematics in 30 elementary schools this year.

Washington and Arlington, the systems where serious labor problems have developed, pay teachers the highest salaries in the area. But in Arlington, where salaries now average about $18,500 a year, teachers complain that recent increases have been far below the rate of inflation.

In Washington, the teachers union has adopted its "work-to-the-rule" job action to try to force the school board to resume contract negotiations and extend its old contract which expired in July.

Negotiations stalled because of a procedural point - how many hours teacher union negotiators can be away from classes without losing pay. School board president Conrad Smith charged that the union wants to avoid serious talks on lengthening the teachers' workday, which is one hour shorter than in the suburbs.

Union president William Simons says the board wants to "break the union. They just wish we would dry up and go away," Simons said, "but that's not going to happen."