Republican governors, their ranks thinned by the Nov. 2 midterm election, turned on President Reagan's representatives here today and blamed their losses on the nationwide unemployment problem and administration policies that are "scaring people to death."

South Dakota Gov. William Janklow, a conservative Republican, used that phrase in arguing with White House pollster Richard B. Wirthlin's claim that there is broad public support for Reagan's policies and the Republican Party faces "no problem that can't be solved by a 3 percent growth rate."

The 19 Republican governors gathered here -- almost a third of whom will not be back in office next year -- listed a dozen other concerns, ranging from huge federal budget deficits, excessive military spending and a general perception that "Republicans care only about the rich" to antagonism of minorities and women, tolerance of narrow-focus, negative campaign groups, and what Janklow called "screwball ideas" for changing Social Security.

The public spanking of the administration by the Republican governors marked a political turning point. For the past two years, while some of them had grumbled privately about Reagan's spending priorities, they had strongly supported his efforts to slow the growth of the federal budget, cut taxes and shift programs back to the states.

Richard S. Williamson, the assistant to the president for intergovernmental relations and ranking administration official at the meeting, attributed the critical outburst to "nervousness about the election results," which reduced the number of GOP governors from 23 to 16 and left several of the survivors severely shaken.

"There's a lot of frustration," Williamson said, "and the administration is an easy target."

Williamson began the session with a claim that the election was "a wash." While the president was "very disappointed" with the loss of seven governorships, he said, the attrition rate among Republican governors had been worse in earlier GOP administrations and Reagan's "top priority" of keeping the Senate in Republican hands had been achieved.

Wirthlin followed with a slide presentation in which he argued that unemployment was the "key" to the 1982 losses and "recovery will be the key to victory in 1984 . . . .

"It would be a serious mistake to take it as a repudiation of our Republican ideas," he said. "There is strong support for the vision we hold."

But the governors had been given a far more critical review of the 1982 election in an earlier closed-door session by pollsters Robert Teeter and V. Lance Tarrance and campaign consultant John Deardourff, who emphasized long-term problems they said were facing the Republican Party.

Deardourff, who designed the Republicans' 1982 "stay the course" ad campaign, said he told the governors that "we have very serious perception problems . . . and we're not going to change perceptions until we change policies at the national level."

So the question period quickly turned into a barrage of ill-concealed criticisms of the administration.

Oregon Gov. Victor Atiyeh, reelected despite the recession in his state, asked if Reagan would accept a slowdown in military spending. Williamson replied they should "not expect substantial cuts" in the defense budget.

New Hampshire Gov.-elect John Sununu, the only Republican to beat a Democratic incumbent this year, told Wirthlin, "I don't agree at all" with the White House election analysis, and warned that if the unemployment issue disappears by 1984, Democrats will have others to exploit.

Retiring Gov. Robert D. Ray of Iowa and New Jersey Gov. Thomas Kean raised pointed questions suggesting that administration policies had made it difficult for any Republican to win the votes of blacks and women.

Wirthlin said the "label of being unfair" to those groups had been "tagged on the Republican Party many years ago," but conceded "it's been heightened and focused by what's happened the last two years."

Williamson said "the president appreciates the importance of depolarizing certain constituencies" and will work on the problem.

Vermont Gov. Richard A. Snelling, also reelected, said those constituencies are "very frightened" by the way the administration approaches issues of importance to them. "There is no conservative sentiment [in the country] sufficient to support a conservative government," he said. Wirthlin disputed the assertion.

Indiana's Robert D. Orr, one of the most conservative governors at the table, charged that Reagan's defense of record budget deficits "makes it hard to show the desires of the people are being carried out."

Then Janklow, who won a handsome reelection victory this month and is considering a possible 1984 challenge to Sen. Larry Pressler (R-S.D.), launched into a lengthy speech, ridiculing "screwball ideas" the administration offered to reduce Social Security spending, its defense of budget deficits and its rationalizations of what he said were tacit alliances with organizations like the National Conservative Political Action Committee (NCPAC).

"We talk about sensitivity," Janklow concluded, "but we don't show sensitivity to people's feelings about these issues."

Governors who in the past have been quick to defend Reagan and his policies, like Tennessee's Lamar Alexander and Pennsylvania's Richard L. Thornburgh, both reelected, and Wisconsin's retiring Lee S. Dreyfus, either joined in the criticism or peppered Wirthlin with skeptical questions.

No one except Wirthlin and Williamson gave what could be characterized as a defense of the president.

Even Illinois Gov. James R. Thompson, the chairman of the group, who faces a possible recount of his apparent squeaker victory in Illinois, said the voters "don't think we are on the right track or the Democrats are on the right track. They think we're all off-track."