The director of Action, the government's volunteer agency, was nearly two years into the job and frustrated. After decades of spending, he told a reporter, federal programs had become a "bureaucratic nightmare," and it was time for change.

"We need to look to a different scale of action. The place where I see the most imagination and zest is at the neighborhood and local level."

It was the autumn of 1978, and the speaker was Sam Brown, the antiwar activist turned Carter administration official.

In the winter of 1983, a little over four years and an ideological eon later, Brown's successor, Thomas A. Pauken, leaned forward in his chair and spoke with intensity about exorbitant federal spending for low returns, the burdens of the bureaucracy and the need for change.

"We've got to get away from the large social bureaucracy and get back to the social entrepreneurs," he said. In other words, return the programs to the neighborhood level.

Perhaps nowhere else in Washington has the philosophical change at the top been more dramatic than at Action, a tiny agency that grew out of John F. Kennedy's dream of peace, friendship and service and Lyndon B. Johnson's Great Society.

In 1977, President Carter reached into the thrashing bag of New Left movers and shakers and drew forth Brown, the son of a Republican businessman who grew up to become a kingpin of the Vietnam Moratorium and a leader in Eugene McCarthy's "Children's Crusade."

In 1981, President Reagan tapped Pauken, a onetime Democrat who nevertheless describes himself as "old right," a former national chairman of the College Young Republicans and a military intelligence officer who volunteered for Vietnam at the height of antiwar activism.

"The Vietnam experience was at the root of what Pauken is doing," said one former agency employe. "He wanted nothing more than to be the person who followed Sam Brown."

Pauken, who concedes that "it was the one position I was interested in" put it mildly: "There is a certain symbolism here."

But in many respects the game at Action these days is a videotape of one four years ago, with only the colors of the jerseys changed.

Item: Four years ago, Action was investigated by the General Accounting Office. The agency was absolved eventually, but not before its GOP detractors had aired a laundry list of charges, including one that Brown was scheming to divert millions in grants to his left-leaning friends.

Now the GAO is digging through the Action files again, at the request of congressional Democrats who have accused Pauken of "politically motivated treatment" of agency programs and "questionable allocation of agency funds."

Item: In 1978, Brown was under fire from his employes for disruptive reorganizations and for allowing "cliques" of old-line activists to exclude career staff from the decision-making process. His decision to hire some of his former associates, including one of the Chicago Seven defendants, as agency consultants made agency employes skeptical and enraged his political foes.

Former and current employes now complain that Pauken and his political aides have cut career staff out of the decision-making process. And they say he is reorganizing programs out of existence and laying off career employes at the same time that he is larding the agency with $130-a-day consultants and experts who happen to be his political cronies and personal friends.

Some people call it politics, but it's not the kind of politics Pauken says he appreciates. Questioned about the highly paid consultants he has hired, including several Texas friends who were involved in his congressional campaigns, his intensity turned to temper.

"If you're liberal, it's pretty hard to do anything wrong in this town. If you're conservative and they can't get you on incompetence, they'll try for something else," he said. "We have one of the most talented groups of people in town, and I've picked the team."

He also has picked the programs, and by that standard Pauken appears to have gained more ground than his predecessor. Brown's Urban Volunteer Corps, envisioned as a support program for President Carter's package of aid for the nation's cities, never got off the ground.

Neither did his Good Neighbor Fund, seen as a way to provide "seed money" to demonstration projects that would develop the momentum to carry on without federal help.

Pauken's two initiatives--the Vietnam Veterans Leadership Program and the Young Volunteers in Action--have fared better. The VVLP, a combination employment counseling service and public relations effort for Vietnam veterans, is operating in 38 cities, and Action hopes to move into 17 more.

YVA, which involved 3,300 teen-aged volunteers in a variety of community programs in its first year, is expected to grow to 200 projects and 15,000 volunteers this year.

Both programs involve the "seed money" concept. Federal grants to the YVA projects are limited to two years; veterans' projects will be cut off at the close of fiscal 1984.

What's happening at Action, Pauken says, is "as good an example as any in a dramatic change in philosophy . . . . We're developing an alternative social strategy. I see the genesis of an approach, a model that works.

"You can make a difference. You can move the bureaucracy."

Maybe so. But in 1978, Sam Brown said the same thing.