Yale University president A. Bartlett Giamatti last night called on the nation's colleges to give major assistance to high schools because of the higher institutions' "deep self-interest" in getting better students.

In a lecture at the University of Maryland, Giamatti strongly rejected the idea that college academic quality has declined because of the vast increase in students enrolled over the past two decades.

He said the decline stemmed from social and intellectual trends. But he said the "devaluation of standards" can best be corrected by colleges working directly to improve teaching and curriculums in high schools.

Giamatti told the audience of 250, which included a number of college presidents from this area, that Yale already is trying to do this in the vicinity of its campus in New Haven. Although similar programs are under way elsewhere, Giamatti said they are not nearly widespread enough.

He said that this could be done without "a lot of federal money," but that some assistance to "free up teachers in the summer" to attend college courses would be helpful.

Giamatti, a professor of English who has been president of Yale since 1978, was a member of the National Commission on Excellence in Education, whose recent report decried "a rising tide of mediocrity" in the nation's schools.

The report called for higher academic standards, but, responding to questions after his lecture, Giamatti said there "probably is nothing" the federal government can do to promote this. Giamatti said, however, that he believes the federal government has an obligation to increase the number of students who can afford to go to college, to aid research, and to enforce constitutional guarantees of equal opportunity.

Taking issue with President Reagan's assertion that the federal role in education should be small, Giamatti said that government has played an important part in promoting schools and colleges since the late 18th century. He noted that money from the sale of federal lands was earmarked to aid education and that specific programs for agricultural research began almost a century ago. Both federal research money and financial aid for college students have increased enormously, Giamatti said, over the past few decades.

"The results have varied," he said. "Institutions have expanded. They have become dependent. They have improved . . . . Fear of federal control has spread much faster than the control has. Enormous benefits to the nation and the world have accrued . . . . "

When pressed by suggestions from the audience that the federal government should do more to raise the quality of the colleges it assists, Giamatti said: "If the institutions don't do that themselves, then there is no way" to do it. "There probably is nothing the federal government can do to instill an instinct for quality," Giamatti said, "except to work on itself."

He also opposed any federally sponsored exam or any minimum test score to qualify students for federal aid. "I have no conviction at all that you will do anything to raise standards because someone is using a test," Giamatti said.

Although he criticized Reagan administration proposals to cut education spending in his speech, Giamatti told a reporter later that he strongly favored merit pay for teachers, a plan backed by President Reagan that is included in the Excellence Commision report.

Although the National Education Association opposes the idea as unfair and unworkable, Giamatti said, "The teachers have a fundamental problem because on the one hand they wish for the attention they deserve and on the other hand they object to what millions of people already go through, undergoing evaluation in order to get additional reward."

But Giamatti added that the general level of teacher pay should be raised along with extra money based on merit.