Maryland Gov. Harry Hughes' call for a special session of the General Assembly next week to try to offset federal cuts in unemployment benefits appears to be just what President Reagan had in mind when he proposed increased state responsibility for social programs and reduced federal spending.

But the Maryland-style New Federalism is causing unexpected problems for Republicans in the state. It has sparked a chorus of protests against the president's policies and benefited Reagan's strongest detractors--Democrats and labor unions.

Nonetheless, Republican leaders are supporting Hughes' request for $9 million in state funds to make up for the federal reductions, saying the state's involvement is a sign that New Federalism is working.

"What's happening should be happening," said Senate Minority Leader Edward J. Mason of Allegany County, who quickly agreed with Hughes that a special legislative session was necessary. "The legislature has to respond to a state problem. Unemployment is a state problem."

But the unemployment compensation issue, in which 11,000 unemployed Marylanders are scheduled to lose federal checks after this week, has placed Republicans in a tight political bind. At the special session next week, they will be in a position of supporting a Democratic governor in an election year and approving legislation that the state's politically powerful labor unions urged Hughes to propose. They will have to listen as Democrats attack Reagan for insensitivity to the unemployed. They will have to admit that, even though Reagan has promised an improvement in the economy by next year, the fiscal picture in Maryland is not so rosy.

Republican leaders have also said they will support legislation that Hughes will propose next year--if he is reelected--that would increase the contributions that employers and businesses must make to the state's unemployment insurance trust fund. The fund, the sole source of money for unemployment compensation, has $275 million in it now, but is expected to run out next summer.

The business community, which opposed legislation to increase benefits for the unemployed during the last legislative session, is also in an awkward bind. But it too supports the special General Assembly session, accepting that Hughes' prolabor legislation is simply "a fact of life."

"It bothers me, but I'm not hard-hearted enough to say we don't have to deal with this problem," said Jack Mosely, head of U.S. Fidelity & Guaranty Co. in Baltimore. He also sounded the Republican theme: "It's a Maryland problem, not a national problem. I wholeheartedly agree with Reagan putting this kind of program back at the states."

Ernie Honig Kent, a lobbyist for the Maryland Chamber of Commerce, said her group also supports Hughes' action, but not without worries.

"We recognize that people are falling through the cracks and we understand what the governor is trying to do," she said. "We're just concerned that the session might get out of hand and labor might get more than what they say they want."

Even Republican gubernatorial candidate Robert A. Pascal, the Anne Arundel County executive, is caught on this issue. Pascal, who is competing for coveted labor support, delivered a stinging letter to the White House last week, asking Reagan to restore benefits for the unemployed.

"The national economic picture may be brightening," Pascal wrote to Reagan, "but there remain very serious levels of unemployment, especially in the heavy industrial states such as Maryland."

While Republicans and business approach the issue timidly, Hughes and Democrats are making political hay.

"It's another example that the 'safety net' has a lot of gaping holes," Hughes said of the new federal formulas that were responsible for the cuts in Marylanders' unemployment benefits. "As a matter of fact, I'm not sure the net exists."

Hughes and his allies dispute suggestions that the state takeover of extended unemployment benefits is a sign that Reagan's programs are successful.

"The governor has said this is by no means long-term," said Hughes' press secretary Lou Panos. "The state can't continue to do without it federal aid indefinitely. They Reagan administration are making them the unemployed sacrificial lambs on the altar of a balanced budget."

Hughes has been receptive to increased state responsibility for some federal education and transportation programs, but he opposes the key component of New Federalism, which involves a federal-state exchange of welfare and Medicaid programs. State officials estimate that the swap would cost Maryland $16 million this year.

"This is a graphic case in point that the supposed New Federalism is really more closely akin to the old trickle-down system," Panos said.