The ability of American high school students to solve complicated math problems remains "unacceptably low" despite significant improvements in computation skills among younger students, a national study reported yesterday.

Students in the bottom quarter of those tested have made substantial gains, the study said, but scores for the top quarter have stayed virtually the same, or slipped slightly, since 1978. The pattern of the results, with improvement in fundamental skills but declines or stagnation in reasoning and analytical abilities, is similar to those reported over the past few years in reading and science test results.

Beverly L. Anderson, director of the federally financed National Assessment of Educational Progress, which conducted the study, said the data probably indicate that the back-to-basics movement and the widespread adoption of minimum competency tests have succeeded in improving basic skills. But Anderson added that students' difficulties with more complicated analytical tasks was "disquieting."

She noted, though, that from 1973 to 1978 both computation and analytical skills had declined.

The new survey, financed by the U.S. Department of Education, was based on a nationwide sample of 46,000 students in public and private schools. Results were given for 9-year-olds, who are generally in fourth grade; 13-year-olds, generally in 8th grade, and 17-year-olds, who are mostly high school juniors.

Overall, the study showed substantial gains for the 13-year-old group, no changes for the 9-year-olds and a slight decline for the 17-year-olds, particularly in exercises that require students to demonstrate an understanding of math knowledge and apply it to non-routine situations. All the improvement, Anderson said, came on "more routine skills, such as computing, recognizing geometric figures and answering simple, one-step 'story problems.' "

"I'm afraid this suggests the schools are emphasizing . . . paper-and-pencil computation and the storage of isolated facts in the head," said Shirley Hill, a professor of mathematics and education at the University of Missouri.

Hill said the spread of computers has reduced the importance of such abilities and has increased the demand for "logical and analytic skills," which schools are not teaching well.

The report said the proportion of high school students who have studied computer programming doubled to 10 percent between 1978 and 1982. The proportion of junior high youngsters with access to a computer terminal in school, it said, climbed from 12 to 23 percent.

Although the achievement gap between whites and blacks has narrowed, the report indicates that it is still substantial and increases as students get older. Among 17-year-olds, the report said, 59 percent of blacks are in the bottom quarter nationwide, compared to 18 percent of whites and 49 percent of Hispanics.

The study said the blacks' gains were greatest in basic computation and occurred mostly in predominantly black schools in "disadvantaged urban areas."