The White House, looking toward the 1984 election, is beginning for the first time to voice public concern about two groups considered vital to the fortunes of Ronald Reagan and the Republican Party: blue-collar workers and women.

With the announcement that the national unemployment rate had climbed to 10.8 percent, Richard Wirthlin, Reagan's pollster, said yesterday that the president will not find it easy to rebuild the coalition of blue-collar Democrats who were essential to his 1980 victory.

These voters are "bruised and frightened," Wirthlin told a breakfast meeting of reporters. "A large number of swing voters who flirted with the Republican Party have now gone back to the old banners."

"One of the challenges politically is to get these voters back," he added. "It can be done, but it's not going to be done easily or quickly, and it has to be done with some improvement in the economy."

Meanwhile, an internal White House report on "the gender gap" is circulating among women's groups that reveals a far deeper concern about Reagan's problems with women voters than has previously been acknowledged. The report is written in a dry, analytical tone, the way, one feminist said, "a bureaucrat always tells the bad news to the boss."

But the conclusions of the 12-page report, prepared for the White House Coordinating Council on Women, contradict almost everything White House strategists have said in public about the differences on how men and women feel about Reagan.

The report maintains that the gender gap "is not new, and not necessarily bad." But it notes that a majority of women voted for Democratic congressional candidates in 1982, and this trend "could prove dangerous for Republicans in 1984."

"New, bold and creative ideas are necessary to deal with the gender gap," says the report, written by Ronald H. Hinckley, a special assistant to the director of the Office of Planning and Evaluation. To correct problems with some groups of women, the report recommends "communications plans" aimed at countering the opinion among many women that "Reagan is likely to start an unnecessary war," and that he has not slowed inflation.

But the report, based on a study of findings in 20,000 interviews by Wirthlin's firm, says other problems cannot be solved by better public relations partly because of dramatic changes in American lifestyles during the 1970s. The number of families headed by single women, for example, doubled to 6.6 million in the decade, and "a disproportionate number" of these women are poor, black and on welfare.

"Fear of losing government benefits appears to be causing women to oppose the administration," the report says at one point, adding later that separated and divorced women feel "Reagan is a threat to the supports originating with government." The report recommends several proposals to overcome these feelings. One calls for a program to end income taxes for the working poor (those earning less than $12,000 annually). Another suggests changes in welfare programs to keep fathers from abandoning families.

The White House has also been grappling with ways to win back blue-collar voters. But Wirthlin said yesterday that the uproar over the idea of taxing unemployment benefits damaged this effort.

"That was a blunder," the pollster said of the now-abandoned study option, although taken alone it "doesn't shatter" the Reagan coalition.

"The thing that will bring the blue-collar voter back is a rising standard of living and a reduction in unemployment," Wirthlin said, adding that the administration would also need to "signal" that it "cares" about the jobless.