More than a third of Montgomery County ninth graders have failed a state mathematics proficiency exam, a showing that school officials are calling perplexing in a county where student test scores traditionally rank among the highest in the nation.

Surprised officials said yesterday that only 65 percent of the students who took the test, administered for the first time last fall, passed. The officials also pointed out that those results compare favorably with figures released last week showing that statewide only 40 percent of the 50,000 Maryland ninth graders passed the exam.

Prince George's County scores were not available.

The test, which measures students' ability to compute simple mathematical problems, may be made a requirement for high school graduation eventually.

Schools officials in Montgomery said yesterday the poor showing, coming at the same time as a wide-ranging national debate is under way about the quality of science and math programs in public schools, could generate pressure for curriculum changes.

"Year after year, you get used to seeing scores in a certain pattern and then when something like this happens you get extremely surprised," said Stephen Frankel, director of the school system's testing department. "Why suddenly the results of this test are so low when in other tests our students score better than any large school system in the nation doesn't make sense. But one of the things we are going to be looking at over the next month is whether there may be problem areas that we don't know about."

Frankel said school officials also will try to ascertain whether the exam had any defects contributing to the poor showing and whether the percentage of questions that had to be answered correctly to pass--80 percent--may have been too high. As eighth graders, the same group of county students took the California Achievement Tests last year, and the average math score was in the 78th percentile.

Before the test results became available, some Montgomery school officials described the state math proficiency test as "simple" and said they would be surprised if most students did not do well on it. The exam was designed to test the ability to add, subtract, multiply and divide whole numbers, fractions and decimals; change decimals into fractions and percentages; find the area and perimeter of simple polygons, read graphs and solve simple formulas.

Among questions given as samples in a guide for teachers that accompanied the test were these:

Jane's basketball scores for February are: 10, 24, 6, 12, 18. Find Jane's average score. Answers: a)70 b)35 c)12 and d)14. Correct answer is (d).

Find the missing term: 6/8=N/16. Answers a)2 b)96 c)12 d)3. Correct answer is (c).

A roast takes 2 hours and 10 minutes to prepare. If dinner is at 5:30 p.m what time should it be put in the oven? Answers a) 3:40, b) 7:40 c) 3:20 d) 7:20. Correct Answer is (c).

The mathematics test, along with a proposed citizenship and functional writing tests, are part of a statewide effort, begun in 1976, to stiffen graduation requirements and to ensure that state-approved curriculums are being used in local school districts. In 1982, for the first time, students were required to pass a functional reading test before obtaining a high school diploma.

State officials said the initial poor showing on the math test is similar to the showing on the reading test when it was first administered in 1976. At that time, 67 percent of the ninth graders failed it. Now, more than 95 percent of the 12th graders in the state have been able to pass it before graduation.

Montgomery officials said their math results are much worse than scores from any previous reading tests. The highest failure rate for that test in the county has only been 11 percent, said James Myerberg, coordinator of testing in the county.

School officials said if the math test is required for graduation, students will take the test in ninth grade, but most likely will have up to nine chances over four years to pass it.