The Reagan administration illegally has impounded part of the money voted by Congress in December for library services, according to a General Accounting Office opinion letter sent to Rep. Peter A. Peyser (D-N.Y.) yesterday.

The dispute involves the Library Services and Construction Act. Congress, in its final money bill last December, voted grants for Titles I and III that totaled $71.5 million annually, according to the New York State Education Department.

But the U.S. Department of Education doled out funds for the first quarter of 1982 at an annual rate of about $52 million, the figure President Reagan requested in September.

On the strength of the GAO letter, a spokesman for the New York State department said it had requested the state attorney general to bring suit for more than $3 million that the state would lose if the impoundment were allowed to stand.

In addition, aides to House Education and Labor Committee Chairman Carl D. Perkins (D-Ky.) were reviewing the possibility that there were other illegal impoundments.

The GAO opinion, signed by general counsel Milton J. Socolar, hinged on a mandatory allotment provision in the library law, requiring that all money provided by Congress be given to the states.

Socolar said that the executive branch has no power under any provision of the Budget and Impoundment Control Act to withhold funds if the program operates under a mandatory allotment law. This is true, the opinion said, even when the president requests rescission of the money and is awaiting congressional action.

Ed Dale, spokesman for the Office of Management and Budget, said a rescission was sent up yesterday.

Dale said that in the past the OMB has operated under the theory that when a rescission has been sent up or is about to be sent up the executive can withhold the money legally until Congress acts.

He said the OMB is studying the GAO opinion.