When Virginia Gov. John N. Dalton asked Dr. Jean Harris, a former Washington physician, to join his cabinet three years ago, Harris was somewhat skeptical.

Being black, a woman and a Democrat, she wasn't sure what role she would play as the first woman and black ever to serve in a Virginia governor's cabinet. Furthermore, Republican Dalton was asking her to take a $26,000 cut in pay.

Some Virginia civil rights activists and black politicians were unsure about the arrangement too, and some of them still are, critizing Harris for adopting what they see as a low profile in the state administration.

Harris, a young-looking woman of 49, says she accepted the job as secretary of human resources because of its challenge and because she was a first. "I think there was a lot of skepticism, and especially on questions of my race, sex or even about my knowledge," Harris said recently in spacious office in the Ninth Street Office Building across from the State Capitol here.

Today, as the $59,000-a-year administrator of one of the largest division of Virginia government, in which she supervises 15 agencies, some 22,000 employes and a two-year budget totalling $2.3 billion, Jean Harris -- a Richmond native, wife and mother of three -- says she feels she has answered many of the skeptics, including herself. She beams when she talks about the role she was somewhat reluctant to accept:

"I love it. It has been a tremendous experience -- [state government] is where the action is," she said emphatically. "This is the attainment of an opportunity to see whether or not it was really possible to move government . . . and it is."

Others who have watched the Dalton administration closely agree that Harris has been both successful and enthusiastic in her job. They say she has been a responsive, solid administrator. Yet, there are a number who question her visibility in the state government, among them American Civil Liberties Union lobbyist Judy Goldberg.

"I haven't really seen her take enough public positions on the issues," Goldberg said. "I don't know how she's applied her experiences of being black and a woman to the job. I'm not aware of any innovative or aggressive changes that she's made."

Virginia NAACP chairman Jack Gravely said "I wish she would have taken a higher profile. A lot of blacks don't know she is black." His group hasn't worked that close with Harris, he said, but "the record will reflect that she's been very successful."

State Sen. L. Douglas Wilder from Richmond, who grew up with Harris and attended college with her at Virginia Union, agreed that she's probably not recognized as a black cabinet member: "You don't think of her as a black in the cabinet." Wilder, the only black in the 40-member State Senate, also said Harris "doesn't take a hard line with being Republican or Democrat."

Harris dismisses questions about her visibility: "The course I've taken is the moddle one where you present alternatives," she said. "You don't win fights by going public.

"We [her department] have won some major battles -- the majority of them -- and many of them have been won from the inside. I think I have a great deal of visibility in program issues, but not in politics."

State Del. Bobby Scott (D-Newport News), one of four blacks in the 100-member House, doesn't think visibility is important to people "who understand the inner workings of state government." She must "bear the administration trademark," he said, rather than embark on her own philosophy.

"She has her own views and people know what her views are," Scott said. "Her views are significantly more progressive [than the Dalton administration's] and I don't think there's any question about it."

But fending off critics who say she is not visible enough has not been the Secretary's main worry. For more than a year, Harris has been locked in a controversy with lobbyist Goldberg and the ACLU over two major issues.

One of them has turned into a class-action suit against Harris, Human Resources, Dalton and several other state officials and state hospitals. That suit, filed in December and still pending, seeks medical and psychological treatment for the 8,300 mental health patients who were sterilized by the state between 1927 and 1973, often without their knowledge or approval.

The ACLU also has attacked Harris for circulating a memo in the Department of Mental Health and Rehabilitation last September asking employes who are members of organizations that disagree with Dalton administration philosophy to disassociate themselves from those groups.

A staunch loyalist, Harris said: "If you accept a role and a responsibility in a particular philosophy, you either support it or not support it. The question is whether or not you should step out and take a more public issue."

Harris talks enthusiastically about state government, about politics and about touching the lives of "each and every citizen of the commonwealth." But until her appointment she hadn't considered government work. From 1964 to 1970 she was a successful private physician in Washington and when Dalton selected her, she was a clinical professor at the Medical College of Virginia here.

"I was never politically active," she said. "I think it's just more of a political philosophy that I'm in tune with. I don't view myself as a politician, I'm a program person who sees people and their needs."

Despite her lack of political activity, she says she would campaign actively for Dalton if he could succeed himself. Dalton can't but that doesn't stop Harris from espousing the governor's philosophy about curbing the growth of state government.

She talks favorably of the expansion of state's rights and speaks of reducing government costs and talks about "helping people help themselves." In fact, she quickly dispelled her Democratic label of the past: "I'm an independent."

As Human Resources secretary, Harris said she has sought to change the image of her departments and to bring the bureaucracy closer to the people. "We are gradually changing the image of human resources from welfare to that of a social service . . . the first thing most people think of when they hear human resources is welfare."

Joy Manson, Dalton's senior executive assistant, said Harris has been successful at "bringing the private sector and government together." Dr. James Kenley, Virginia commissioner of health whose agency is under Human Resources, calls Harris "an unusually good bureaucrat."

Harris was nominated by Dalton to receive one of 21 national distinguished service awards presented during the fifth annual National Governors Association meeting in Atlantic City earlier this month.

In a job that she acknowledged has had its surprises, Harris said the biggest surprise has been watching her own attitude about Virgina's conservative reputation change: "For all its labeling as a conservative state, Virginia is far ahead of the others. It's surprised me. We're getting there pretty rapidly."