The nation's governors yesterday formally agreed to begin negotiations with President Reagan on a bipartisan plan for sorting out federal and state responsibilities.

They acted unanimously after Senate Majority Leader Howard H. Baker Jr. (R-Tenn.), describing this as a "now-or-never" opportunity, pledged Senate hearings this spring on the federalism issue and said he would try for a floor vote before adjournment.

House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.), however--describing the sorting-out proposal as "part of the overall retrenchment on social policy" of the Reagan administration--told the governors there would be "hard scrutiny" of the idea in House hearings.

But under prodding from some Democratic governors who advocate the sorting-out process, O'Neill promised that the House "will not take federalism and put it under the table." He even suggested that congressional Democrats might try to draft an alternative of their own.

Thus, the three days of the midwinter governors' meeting produced what Reagan operatives had hoped to obtain: an agreement to negotiate the substantial differences between the original Reagan plan and the governors' traditional position, in an atmosphere where partisan differences do not foredoom the product to automatic defeat in Congress.

White House intergovernmental liaison Richard S. Williamson said negotiations probably would begin next week in hopes of having a draft bill ready by the end of March.

"We're willing to look at different formulas," Williamson said. "We've tried to signal that with all the talk about flexibility."

Reagan has proposed that the federal government take over the federal-state Medicaid program while the states pick up food stamps--now all federal--and the state-federal program of Aid to Families with Dependent Children. Predictably, the governors like the first half but have asked Reagan to set aside the second. Their traditional position has been that the federal government should take over all support programs for the poor.

National Governors' Association Chairman Richard A. Snelling (R-Vt.) said he did not believe it necessary that the governors and White House be in "perfect agreement" on a draft bill.

"In the end the United States Congress will decide what the federalism proposal is going to be like," Snelling told reporters earlier this week, "and it won't be like what the president wants--I'm sure of that. It won't be like what we want--I'm sure of that."

The White House has promised extensive consultations with mayors, state legislators and county officials before sending legislation to Congress, but the administration has not yet made any arrangements with those groups for doing so.

"We understand that it's an open process that they are talking about," said Mike Brown, spokesman for the U.S. Conference of Mayors, whose big city mayor members have been among the most vocal opponents to federalism. "I guess feelings would be hurt if we couldn't be a part of that process."

The final session of the governors' meeting was full of reminders of the partisan and policy differences that threaten federalism and still might sink it in the tricky process ahead.

Democratic governors, at a party caucus preceding the formal session, submerged their endorsement of the initiative in a statement lambasting the Reagan budget and economic policy and warning that "the promises of federalism will ring hollow" unless the economy is cured.

Baker gave it a push forward by pleading that it not become "a political football" and promising to seek a Senate vote this year.

But then O'Neill, after calling Reaganomics a "blueprint for disaster," began throwing cold water on the federalism initiative. He said he could "very easily" agree to returning about 17 of the 43 programs Reagan suggested sending back to the states--mainly in the education and transportation fields.

But he questioned whether any income-support programs should be included in the swap and asked "if a time of recession is a good time" even to be discussing such a switch.

"If we turn these programs over to you," the speaker asked, "can you afford them? Can you handle them?" Pointedly, he told Delaware Gov. Pierre S. duPont IV (R), 47, that if he were old enough to remember the Great Depression, he would know that "local governments couldn't handle the programs. That's why they were turned over to the federal government."

But two pro-federalism Democrats, Arizona Gov. Bruce Babbitt and Colorado Gov. Richard D. Lamm, pleaded with O'Neill not to bury the proposal. "We're not in the '30s anymore," Lamm said, and Democratic governors and local officials, no less than Republicans, "are tired of playing 'Mother, may I' with federal bureaucrats."