While Washington lobbyis Frank R. Wolf isn't worrying about what the federal bureaucracy is about to do to baby food or farm implement dealers, two ofhis clients, he is likely to be worried about "the kindly grandfather."

That's the label Wolf and many Northern Virginia Republicans have given to Rep. Joseph L. Fisher, the two-term Democrat whose strong image and popularity have troubled the GOP ever since he unseated longtime Republican representative Joel T. Broyhill in 1974. Last year Fisher, who is 63 and a granfather, polished off one of the GOP's most popular state delegates, Vincent F. Callahan, by a 10 to 7 vote margin.

Wolf, a conservative Republican who lives in Fairfax County and practices law in Rosslyn, would like to be the person to take on Fisher next year, but not until after what the GOP's 10th District chairman concedes is a negative campaign" against Fisher has run its course. Although the actual race for Fisher's seat is 13 months away, a group called the "10th District Republican "Task Force" has already begun circulating anti-Fisher bumper stickers, flyers, newspaper ads, and letters in Fisher's district.

Not surprisingly one of the key figures behind the drive is Wolf, a former congressional aide and a deputy assistant secretary for congressional relations in the Interior Department during the Nixon and Ford administrations. Wolf carries the title of "task force chairman" and, according to some Republicans, is the only member of the party who is now seeking the party's nomination to take on Fisher.

That, as Wolf himself says, will not be an easy chore and a major reason is the way the white haired Fisher looks. "Mr. Fisher looks anything but liberal - he is very distinguished looking and speaks softly," Wolf complained in an anti-Fisher flyer his group mailed to between 3,000 and 4,000 Northern Virginians. "In fact, he looks like a kindly father, grandfather, or favorite uncle," Wolf, who is 38, groused.

Wolf's task force, which he describes an "an arm" of the GOP's 10th District Committee, has already raised between $2,200 and $2,400 for its operations, opened a small office in Falls Church, and begun an effort to portray Fisher as a free-spending "ultra-liberal" whose views do not reflect the district. In newspaper ads, Fisher has been described as "voting for legislation that forces up the cost of housing" preventing new families from affording "a home in Northern Virginia."

Fisher's aides complain that much of the material incorrectly describes the congressman's voting record and say the rhetoric appears similar in tone that the material used against Fisher last year. "My clear impression is that it is being produced locally by a conservative group that narrowily lost to Callahan in the Republican primary," said John G. Milliden, Fisher's executive assistant.

Wolf, then poorly known in Republican party circles, lost that primary to Callahan, a moderate Republican, but his strength shocked some Republican officials. If he wins the nomination next year, as many expect, these Republicans say it will mark another victory for the increasing strength of conservatives in the Northern Virginia wing of the party. This summer conservatives scored another major victory in southern Fairfax County's state legislative district, winning three of five nominations and toppling an incumbent moderate in the process.

Wolf and 10th District GOP Chairman Edmund L. Walton insist the district nomination isn't locked up and say the task force doesn't exist just for Wolf's benefit. Although he doesn't deny an interest in the race. Wolf points out that he has not announced formally and probably won't until after the November elections. Both Wolf and Walton also say that the anti-Fisher efforts have not detracted from the November elections which are the focus of most political activity in the state. "I don't think you can start too soon on a congressional race," agrees Callahan, currently seeking another two-year term in the statehouse himself. "A congressional race is like a statewide senate or gubernatorial race, it requires an awful lot of planning."

Still, it is surprising that the GOP appears to have targeted Fisher so early, so well in advance of his race and with a "negative" campaign. Neither that nor "a right-wing" campaign will work, Callahan says.

Walton, the GOP district chairman, says the task force's efforts have to be considered "negative" because, "we don't have a candidate yet" and because Fisher's image is the GOP's major problem."You can talk issues all you want to," Callahan says. "People don't vote on that they vote on images."

Fisher's aides say that the popular image of their man as a conservative is, in fact, correct. "The single most critical vote" a congressman can cast is on the federal budget. Milliken says, and on that measure last year Fisher voted no "simply because he believed the level of spending was too high."

Since then he has voted against an agricultural department bill because he believed farm price supports were "too high" and several other appropriation measures, Milliken said "In fact, Fisher in style and substance is a rather moderate man," he says.

Of course, 1978 will be an off-year election and historically the party in the White House has lost seats in Congress in such elections. So one factor certain to face grandfather Fisher next November will be what GOP District Chairman Walton hopes are Jimmy Carter's "negative coattails."