Patricia Roberts Harris, who started out as one of the more controversial members of the Carter administration, said yesterday she is accepting the president's offer to take over the mammoth Department of Health, Education and Welfare.

The new appointment came as a surprise to many of Harris' constitutents in housing and urban affairs. Some of them said yesterday they were just beginning to get used to her as secretary of housing and urban development after the stir caused two years ago by her initial lack of housing expertise and her reputation for abrasiveness.

"We're all surprised up her on the Hill," said a key aide to the House subcommittee on housing and community development. "She had come in cold and was just beginning to get into her stride. Now they've gone and yanked her out of the place."

Carter offered Harris the top HEW job Wednesday. She said yesterday that she accepted the post "after careful consideration" that included discussion with her husband.

Harris, the nation's first black female Cabinet member, was confirmed as HUD secretary January 20, 1977. Her selection by Carter after the 1976 election had been roundly criticized by mayors, urban lobbyists and others who said she did not have the background to run the troubled housing agency with its staff of about 17,000 and annual budget approaching $11 billion.

At HEW, Harris would oversee a budget of about $199.43 billion and 158,364 employees.

Harris had been a U.S. ambassador to Luxembourg, a ranking Democratic Party official, a short-lived university administrator, a member of the boards of several corporations, including Chase Manhattan Bank, and a respected Washington lawyer. Her housing credentials were scanty.

But during her 30-month tenure as HUD secretary, Harris managed to turn many of her doubters into strong supporters.

"She learned very fast. She appointed very good people under her," siad John Gunther, executive director of the United States Conference of Mayors, one of Harris' ex-detractors.

"Pat's done a good job as HUD secretary. She's certainly shown that she could run the department," said Randy Arndt, spokesman for the National League of Cities.

"Harris comes off pretty good," said another top urban lobbyist who requested anonymity. "Hamilton Jordan and the rest of the "Georgian Mafia" may not liker her because she's kind of abrasive and independent. But they can't toucher her politically because she's black and a woman," the lobbyists said.

Others said that though Harris has earned a reputation as a blunt spokesman for her beliefs, she has managed to stay in Carters good graces because of her loyalty to the administration.

"As an independent black, she could have strayed off the reservation with her disagreements and found support. But she's never done that," said a House Banking Committee source.

Ironically, one of Harris' major administration squabbles was with Jospeph Califano supposedly backed an Office of Management and Budget proposal in the spring of 1977 that, critics said, would have transferred about $4.9 billion from HUD's budget to HEW.

Harris trumped Califano, who later denied having "my greedy paws out for some of HUD's money."

Yesterday, Harris called Califano her "longtime friend and colleague." She said she plans take over his operation after Labor Day.

As to what her plans would be after she is in place at HEW, Harris said: "I would rather not make predictions today about a policy course. I would rather do that in a more orderly way."

However, Harris warned that no one should think that she would be less tough than Califano was in his anti-smoking drive and in his attempts to desegregate universities and colleges in North Carolina.

"Nobody should take in comfort with my appointment in respect to may chaning any policies," she said.