Gregory Holman is assigned to the fair housing office at the Housing and Urban Development Department, but the clutter that covers the desk in his windowless fifth-floor cubicle doesn't have much to do with housing.

It has to do with unions, specifically the American Federation of Government Employes (AFGE). Holman, a shop steward for AFGE's Local No. 476 at HUD, spends most of his time these days doing battle with his employer.

Holman and Local 476 are at the leading edge of what AFGE officials say is a rising militancy among federal employes, an attitude sparked by extensive RIFs, fueled by furloughs and expressed in increasingly bitter confrontations over administration policies.

"We're getting a reputation as getting more active," Holman said in an interview last week. "It's just a matter of survival. We have to be active or we get steamrollered."

The simmering feud at HUD boiled over yesterday in the office of the department's chief negotiator, Hillard Harrison. Holman and Harrison nearly came to blows in what union officials say was an "unprovoked attack" by Harrison, and Harrison says was an "overreaction" on the part of both men. [Details in the Federal Diary, Page C2.]

The union immediately filed a letter of protest demanding Harrison's dismissal. Harrison says he would hate to see the incident "stand in the way of the work we still have to deal with."

For nearly a year, Local 476 has been engaged in an acrimonious fight over a planned RIF at HUD that the union says is politically motivated, discriminatory and unjustified, given that the department is already 400 employes under its personnel ceiling.

Only 300 of the 3,000 eligible headquarters employes are dues-paying AFGE members, but the local has attracted attention far beyond its numbers with rallies, informational pickets and a barrage of grievances. And it caught the ear of Capitol Hill when it charged that the RIFs would impede the department's ability to enforce the law and carry out the will of Congress.

AFGE officials say that two-thirds of the RIFs are targeted toward HUD's priority program areas -- housing and community development. In addition, the office of fair housing and equal opportunity, the department's civil rights arm, would lose 15 percent of its staff, leaving it with fewer people than Secretary Samuel R. Pierce Jr.'s immediate office staff.

Administration officials contend that the fight at HUD is inspired less by a desire to protect union members than to thwart broader policy changes under the Reagan administration.

"They are at odds with the policy," said Office of Personnel Management spokesman Pat Korten. "There's a fundamental difference in philosophy, and they are trying to take it out on the secretary and his team."

AFGE officials don't dispute Korten's assessment, but they say it is their responsibility as public servants to fight policies as well as layoffs. The administration "doesn't give a damn about public housing," says John Mulholland, a labor-management relations official at the union's national headquarters here. "We don't like what management's doing, so we're protecting the public's interest as well as that of our members. What else would we do?"

The unions are finding some members of Congress, particularly those with sizable numbers of federal employes in their districts, sympathetic to their complaints. But other members have found the union charges a convenient hook upon which to hang a broader attack on the administration's housing policies.

In a hearing last week, Rep. Henry B. Gonzalez (D-Tex.), chairman of the House Banking subcommittee on housing and community development, told Pierce that a planned field reorganization, under which most of the RIFs would be conducted, will "wipe out whatever forward momentum is remaining" in HUD's programs and basic missions.

Rep. Parren D. Mitchell (D-Md.) lectured Pierce on the proposed cutbacks in HUD's equal opportunity office, calling it "a pattern that extends itself across all the agencies."

AFGE has maintained that staff reductions to meet budget targets can be handled through normal attrition. But Pierce told the subcommittee that the RIFs were necessary to correct "staffing imbalances" that have grown through years of attrition and hiring freezes.

HUD's critics say that is a fancy way of justifying a plan to put more politically responsive managers in policy-making positions. Pierce acknowledged that he thinks making the decision-making process more political "is just sheerly good management."

It wasn't always that way at HUD. In its entire pre-Reagan history there, it filed only one unfair labor practices complaint. Holman recalls that the union once voted to make Patricia Roberts Harris, Jimmy Carter's first HUD secretary, an honorary member.

But in the past 18 months, the local has filed 43 complaints of unfair labor practices -- the first stemming from an incident on Jan. 22, 1981, two days after President Reagan's inauguration. Local president Ernest Parker says the union will file another complaining about the incident in Harrison's office yesterday.

OPM's Korten says the record of complaints at HUD are "a direct function of the radicalism of that local. They're nasty folks."

Union leaders acknowledge that the uncertainties created by the RIF process are a main selling point to potential members of a union that is forbidden by law from striking and cannot bargain for wages and benefits -- the meat and potatoes of other union negotiators.

Of nearly three-quarters of a million federal employes covered under AFGE contracts, the union counts only slightly more than a third as dues-paying union members. But in the last four months, according to Mulholland, the union has shown a net gain of 3,500 members.

"What's happened is because of RIFs," said Rene Dubose, a national AFGE organizer. "They've shaken the complacency of the federal workers, who are turning to the union for leadership."

At HUD, the union says it has had some effect on the scheduled RIFs--if not to halt them, at least to reduce the number of employes affected. The original plan envisioned a RIF of more than 800. Last week, Pierce and his assistant secretary for administration, Judith L. Tardy, indicated the RIFs would affect about 260 employes in the field and 100 in Washington.

Korten says AFGE is simply "playing the RIFs for all they are worth" in an effort to attract new members. "If I were a union organizer, I'd do the same thing," he says.

On that point, Korten finds agreement in the union. "It's hard to organize a benevolent employer," says Mulholland. "Our best organizer is sitting in the White House."