When a delegation of antipoverty workers recently met with Virginia's Human Resources Secretary Joseph L. Fisher to protest a proposed program that would put some welfare recipients to work, they expected the former liberal congressman from Northern Virginia to agree with their claim that a street-sweeping job is demeaning.

"Well, now, I'm an old sweeper myself," said Fisher, recalling the custodial work he did as a World War II Army private. "It's not so easy to sweep and do it well. I wouldn't support demeaning work, but I don't think there's anything demeaning about sweeping," he said, fixing the startled group with a steely gaze.

"He's from the old school, I guess," attorney Rick Cagan of the Virginia Poverty Law Center said later. "We may have a more receptive audience in the secretary's office than before, but I think it was wrong for anyone to assume Joe Fisher is a knee-jerk liberal."

That is precisely the image that many members of Virginia's conservative establishment had of the 69-year-old Arlington Democrat, who represented the 10th Congressional District for three terms. But in the 18 months since Gov. Charles S. Robb appointed him to the $61,360-a-year cabinet post, Fisher has accomplished what some said was impossible. He has largely succeeded in mollifying conservatives and powerful special interests while pleasing activist advocacy groups whose members say they were ignored by Robb's Republican predecessors.

As human resources secretary, Fisher holds one of the toughest and most politically sensitive jobs in the six-member state cabinet, overseeing a $3.5 million budget and 20,000 employes scattered throughout 15 agencies. A former member of the President's Council of Economic Advisers, he inherited a chunk of the state's bureaucracy that was hit hard by the Reagan administration's federal budget cuts.

Fisher's first task was to determine how to eradicate a multimillion-dollar state Medicaid deficit without infuriating powerful special interests such as the nursing home industry or social welfare groups such as the Poverty Law Center.

"No one else could have undertaken the $123 million Medicaid reduction with so little public outcry," said Robb, who praises Fisher's solution -- a series of benefit cuts and sharply curtailed reimbursements to nursing homes--that left most groups reasonably satisfied that they at least had been consulted about the cuts before they were announced.

That cautious, deliberate style reminds many of Fisher's boss. "Like Chuck, Joe is very controlled," said John Milliken, vice chairman of the Arlington County Board and a former Fisher congressional aide. "Joe is a very orderly, literal person with a strong sense of what's appropriate."

Still, Fisher's decision to join Robb in Richmond surprised many of his friends, who wondered how an erudite native of New England, educated at Harvard and the London School of Economics, would fare in the old capital of the Confederacy.

"At first I was afraid he might be bored, I just couldn't see him fitting in down there working for Chuck Robb," said Arlington Sheriff James A. Gondles Jr. Gondles' fears were allayed last summer when he spotted Fisher striding through the marble halls of the State Capitol in what he called "full Virginia regalia": pin-striped suit, white shirt, tiny cardinal lapel pin in the shape of the state bird and blue tie embossed with the state seal.

"I burst out laughing. He looked just like a Virginia gentleman," said Gondles.

In Richmond, Fisher's unfailingly polite demeanor and his cautious style, coupled with his fiscal conservatism--what Fisher calls "my New England Puritanism"--have elicited praise from unexpected quarters.

"Joe's a fine gentleman doing a superb job," said Sen. Edward E. Willey of Richmond, the often-irascible patriarch who chairs the Senate Finance Committee. "He's not like a lot of them up there in Northern Virginia who advocate living off the public trough. Of course, he's a lot more naturally liberal than a lot of us down here, so his philosophy has to be looked at pretty carefully."

Fisher's laconic style has caused him some problems here. Some observers speculate that he is bored, a belief inspired by his short-lived, unsuccessful bid last year for the Democratic nomination for the U.S. Senate seat being vacated by Sen. Harry F. Byrd Jr.

Fisher denies that he is tired of the cabinet, but he admits that he is toying with the idea of challenging Sen. John W. Warner when Warner's Senate seat comes up for election next fall.

Several months ago, after Fisher stumbled during a budget briefing, Wayne Anderson, Robb's influential secretary of administration and finance, approached him.

"I went to him as a friend and told him what the scuttlebutt in the halls was, which was that he was just too scholarly," said Anderson, a former Alexandria city manager who had known Fisher when he served on the Arlington County Board, Metro Board and the Metropolitan Washington Council of Governments.

"He said, 'Well, when people get to know me better, they'll know I'm not too wise to act.' I think that image of him has totally changed now, but there was concern at the time," Anderson said.

"Joe is not a detail person. He's accustomed to dealing with the big picture, the policy thrusts," said his deputy, Lelia Hopper, 34. A highly respected lawyer, Hopper spent eight years in the General Assembly drafting key pieces of social welfare legislation. "In his job, you can just get buried in detail," she added.

Fisher's cerebral approach was reflected during a recent meeting in his office, decorated with watercolors painted by Peggy Fisher, his wife of 41 years. For more than an hour, Fisher, Assistant Attorney General Julia Krebs, Virginia's Medicaid director and Fisher's two aides debated various responses to a controversial Reagan administration proposal to require some adults to pay their parents' nursing home costs.

As the group disbanded, Fisher gathered his papers and said with a satisfied nod, "I like this problem." When a baffled-looking Krebs asked what he meant, Fisher replied, "You know, it's a good problem, it's intellectually interesting."

"Joe's intellectual ability is central to him," said Bill Fisher, 31, the fourth of Fisher's seven children, a diverse group that includes a rock guitarist, a dancer, two ex-Peace Corps volunteers and a Southeast Asian expert for The World Bank. "He fits things into a worldly perspective that's almost religious." In fact, beginning in 1965, Fisher served as worldwide head of the Unitarian Church for 12 years--"longer than anyone in history," he says proudly.

"Joe's a very, very collected person, much different from anyone I know, so much that you're tempted to think it's extreme," said Bill Fisher, a policy analyst for the Agriculture Department's food stamp program. All the Fisher children call their father by his first name because "he's just more comfortable with that," said his son.

Fisher's personality, according to those who know him well, was shaped by several factors: the New England tradition of public service, his intellectual and athletic abilities, a love of nature and Unitarianism, with its emphasis on tolerance and social activism.

"When we were climbing mountains, my father always told my brothers not to leap from boulder to boulder, but to climb slowly and deliberately, one rock at a time," said Betsy Fisher, 28, a modern dancer with a New York-based troupe. "That's his approach to life."

The first child and only son of a construction company owner, Fisher grew up near Pawtucket, R.I., and spent summers in Cherryfield, Maine, in a rustic family lodge that still lacks electricity and running water.

A natural athlete, Fisher taught himself tennis at the age of 4, and toured European capitals playing in tennis tournaments after his graduation from Bowdoin College in 1935. While an economics graduate student at Harvard, he supported himself as a semiprofessional basketball player and semipro boxer, earning $25 per fight while touring a circuit of gritty New England industrial towns.

After a stint in Alaska as an environmental economist and a three-year Army hitch in the Pacific during World War II, Fisher earned a doctorate in economics from Harvard. He came to Washington in 1947 and worked for Arthur Burns as a senior economist on the President's Council of Economic Advisers during the Truman and Eisenhower administrations.

He remains a ferociously competitive tennis player whose regular partner in Richmond is state Public Safety Secretary Franklin E. White, nearly 30 years his junior. One of the drawbacks of living in Richmond, Fisher has told friends, is his inability to find a tennis club with nondiscriminatory policies.

"I'm quite competitive but not in a pugnacious, belligerent sense," Fisher said recently over lunch at a French restaurant near his office. "It's more a delight in the contest, pitting yourself against somebody. There's nothing like the possibility of getting a stinger on the jaw to keep you awake. That's why I like politics. It sharpens the wits."

Fisher won his first election when he ran for the Arlington County Board in 1964. A decade later, he beat 11-term Republican Rep. Joel T. Broyhill in a stunning post-Watergate upset.

Fisher himself lost his seat to Frank R. Wolf during the 1980 Reagan sweep. His Richmond appointment disappointed many of his Northern Virginia supporters, who were relishing the prospect of a 1982 rematch between Fisher and Wolf. The two have run against each other three times.

"I got to thinking whether I wanted to work really hard just to get back to where I'd already been," Fisher said recently. "I was ready to try something new."

Democratic and Republican leaders generally suggest Fisher is far too liberal to win a statewide race in Virginia. Some of his allies consider his style a potential liability. "From a purely partisan political point of view," said Gondles, "I've seen a lot of politicians with better political sense than Joe."

Fisher says his decision will turn on more esoteric considerations. "If I decide to do it, that will mainly depend on whether I thought I had enough to offer, not whether somebody said I had a 50-50 chance."