Maryland school officials, stunned by the extremely high failure rate of Maryland high school students on a math proficiency exam last fall, predict thousands of high school students may not receive their diplomas in coming years if the passing standard on the test is not lowered.

State officials earlier this month released figures that showed that more than 60 percent of the 50,000 ninth graders in Maryland failed the exam, given for the first time in the state last year.

In Montgomery County, where students typically score among the best in the nation, 35 percent of the ninth graders failed. Scores for Prince George's County students will not be available until this summer, school officials there said.

The test, which measures students' abilities to perform basic mathematical functions, is expected to become a requirement for graduation in four years and is part of a broader effort to ensure that when a student receives a Maryland high school diploma he can perform basic skills.

The disappointing results come at the same time as educators around the country are pushing to raise school standards, which they contend have declined to an unacceptable level. But high school teachers and administrators in counties around the state also suggested there may have been something wrong with the test, devised by the Maryland State Board of Education.

To pass the exam, students were required to answer 80 percent of the questions correctly, but some school officials said that may have been too high.

School officials and teachers warned in interviews that if the test is made a requirement without a subsequent lowering of the passing rate, Maryland may be soon in the same situation as Florida: This year, about 2 percent of all seniors in that state did not receive their diplomas because they failed a required functional literacy test, which included math and literacy skills.

Maryland's 80 percent requirement "was a highly unrealistic standard," said Leonard Granick, testing director of the Baltimore city schools. Only 19 percent of Baltimore's ninth graders passed the math exam. "If the passing rate is maintained, we are going to be facing massive remedial work."

Granick, who has spent the past week analyzing the test results, said students who performed at grade level on national tests had to perform two years above grade level to pass the Maryland math exam. Montgomery officials also said only 25 percent of the ninth-grade students who scored at grade level when they took the California Achievement Test last year passed the state exam.

State school officials disputed Granick's analysis. "I know that's wrong," said Gus Crenson, spokesman for the State Board of Education. "The tests were a long time in the making, and questions, such as Mr. Granick's, were taken up long ago and accounted for."

The tests results show "Johnny can add--he just can't add fractions or measure things accurately," said Paul L. Williams, director of the state's testing department.

The 78-question, multiple-choice test piloted by the state board does not resemble college entrance exams. Instead, it measures students' ability to add, subtract, multiply and divide whole numbers, fractions and decimals. It also tests students' ability to convert fractions and percentages, use data, measure simple polygons and solve simple word problems.

One sample question asks students to find 5 percent of 495. Another asks students to compute how much five pairs of socks would cost if one pair costs 98 cents. Another asks how much change a customer would receive if he paid a $20 bill for a glove that costs $16.85.

In Montgomery, where test results usually draw accolades, school officials said this week they were still searching for reasons why their students scored so poorly. Last year, this year's ninth graders scored in the 79th percentile in math on the Californina Achievement exams; math scores on the Scholastic Aptitude Test continued at a steady 503 points out of 800 after bottoming at 499 in 1975.

"It is clear that the level at which people are expecting kids to perform is not the same as what kids seem to be able to do," said James Myerberg, coordinator of testing for the county. "One of the things we have to look at is why there is this lack of a match."

Some school board members went further than Myerberg's observations and said the poor showing is symbolic of a continuing decline in standards for county students.

"These questions are ones our students should know," said board member Suzanne Peyser, who has pushed for stiffer requirements. "The poor results are just symptomatic of the whole school system."

High school math teachers, however, cautioned against any large-scale assumptions and said that before conclusions can be drawn, changes in student demographics also should be considered.

Montgomery County has experienced a large influx of foreign-born students over the past five years, and many of these students may have had great difficulty in comprehending the instructions, said Michael Kanagy, math coordinator at Walter Johnson High School in Bethesda.

Other high school math teachers concurred and said if the test continues to be administered unchanged, minorities will be the hardest hit. At Montgomery Blair High School in Silver Spring, where minority enrollment is at 60 percent, only 38 percent of the 175 ninth graders passed the exam.

"By far the kids most affected by the the test are kids who have some kind of language problem or who are minority," said Blair math coordinator Allan Graham.

The math exam, along with a proposed citizenship and writing exam, are part of a statewide effort to ensure a standard level of competency among all Maryland students. In 1982, for the first time, students were required to pass a functional reading test before obtaining a high school diploma.

In contrast with the math results, 95 percent of Maryland 12th graders have been able to pass the state reading test. But, as with the math test, school offficials said that when the reading test was first piloted, 67 percent of the ninth graders failed it.

Elsewhere in the country, 37 other states have adopted similar competency exams before diplomas will be granted.

As the public concern continues to mount about the low test scores, Baltimore's Granick said local school officials have been placed in an awkward position.

"How do you tell the public that the standards were very unrealistic?" Granick noted. "If you ask them to bring the number down, you are accused of setting inferior standards. It's a real no man's land."