ith evident nervousness, the nation's governors told the Reagan administration today that they were ready to negotiate a phaseout of federal aid to education, law enforcement and transportation if the federal government in return would take more responsibility for "safety-net" or income-maintenance programs.

The invitation to the administration and Congress to move beyond this year's grant-in-aid reductions into a "sorting-out" of overlapping functions was approved 30 to 5 at the close of the annual meeting of the National Governors Association here.

Some Democrats warned that the Reagan administration was engaged in a "shell game" of dumping problems on the states. And the sponsor of the adopted resolution said he feared that "budgetary tunnel vision" in the administration would short-circuit the effort to realign the federal system.

But Vice President Bush, in a speech delivered just before the final vote, sought to assure the governors that, while the administration is "determined to cut the size, the growth and the cost of the federal leviathan," it "will not allow you to go it alone."

The key language in the resolution drafted by Georgia Gov. George Busbee (D), the outgoing chairman of the association, said:

"The governors reiterate their strong belief that the so-called 'safety-net' programs -- such as welfare and Medicaid -- have been, are, and should continue to be primarily a federal responsibility. We will continue to resist...any effort to shift Medicaid or Aid to Families with Dependent Children in any way...

"We offer to work with the administration and Congress toward a phased-in reduction of federal support in the areas of education, law enforcement and transportation in return for increased federal responsibility for income security programs..."

The governors' resolution thus conflicts in part with the budget cuts pushed through Congress by the administration. Among other things, these reduced federal outlays for Medicaid and other welfare programs.

The resolution was also more general than a straight tradeoff of full federal payment for Medicaid in return for an end to federal aid to education. That idea was put forward for discussion, but not brought to a vote, by Govs. Lamar Alexander (R) of Tennessee and Bruce Babbitt (D) of Arizona.

But criticism from those two governors led to the shelving today of a policy resolution on education drafted by Florida Gov. Bob Graham (D). The resolution, which would have reiterated the need for federal assistance at least for vocational and special education and education research, drew several critical comments and was tabled until the next meeting. Nonetheless, some governors were plainly nervous about opening the door to a federal aid phaseout in the education, transportation and law enforcement fields.

Gov. Brendan Byrne (D) of New Jersey noted that his state's congressional delegation had just fought successfully to keep some federal operating subsidies for mass transit and said he would oppose federal withdrawal from that area. Gov. James B. Hunt (D) of North Carolina urged the governors to be "very careful. If we don't look out," he said, "we're going to snow ourselves. We're serious about sorting out responsibilities, but I haven't seen much indication this administration or Congress is serious...."

The sharpest criticism of the administration's motives came from the two Gov. Browns who were elected today as chairman and vice chairman of the Democratic Governors Association.

Gov. Edmund G. (Jerry) Brown Jr., of California, the new chairman, accused Reagan of playing "a shell game" by using federal budget cuts to "shift the burden to state and local governments."

Gov. John Y. Brown of Kentucky, the vice chairman, said, "Before we fall all over ourselves endorsing the return of responsibilities to the states, we better find out whether we're getting new revenue sources with them...otherwise, there won't be a governor left..."

The president has promised to propose such a diversion of federal revenue sources, but there is no timetable for such a shift to the states.

Even Busbee, who kept the association in line behind Reagan's major budget cuts, said it was time for the governors to speak out against "the alarming tendency so prevalent in the minds of some people in Washington of seeing federalism as a way for the federal government to dispose of its excess fiscal baggage."

Busbee said, "I fear it's only a matter of months before some of our friends in OMB Office of Management and Budget attempt to impose...a sorting-out process guided by one sole criterion: What makes it easiest to balance the federal budgget?

"From the point of view of budgetary tunnel vision," he said, "federalism becomes an easy matter. Just pick out the most expensive, the most difficult to manage, the most politically controversial federal programs, and hand them over to the states and localities with a heartfelt sigh of relief."

Busbee said that "more and more, I'm convinced a major effort will be made in the very near future to transfer responsibility for income maintenance programs to the states...We should serve notice that we will oppose block grants for AFDC or Medicaid."