The Fairfax County Board of Supervisors gave final approval yesterday to a plan that strips the county's embattled housing authority of much of its power and puts the supervisors firmly in control.

"This agreement is a transfer really of where the buck stops . . . in housing," said Supervisor Audrey Moore (D-Annandale). "It's more appropriate for the responsibility to lie here" with the board of supervisors.

Under the new arrangement, the Fairfax County Redevelopment and Housing Authority, which has been largely self-governing, effectively becomes another county agency and turns ultimate control of its spending and programs over to the supervisors.

County Executive J. Hamilton Lambert will have day-to-day responsibility for the authority, and County Attorney David T. Stitt will assume control over its legal affairs.

The board voted 6-to-0 to adopt the plan. Supervisors Martha V. Pennino (D-Centreville), Joseph Alexander (D-Lee) and Republican board chairman John F. Herrity, who is recovering from a mild heart attack, were absent. The housing authority approved the plan last week.

The housing authority and the board have long been political rivals, with housing authority members accusing some supervisors of being less than supportive of public housing for low-income families in the affluent suburban county.

In a conciliatory statement yesterday, however, Supervisor James M. Scott (D-Providence) said it is "not the intent of the plan . . . to undermine the efforts of the housing authority to provide housing for people of low and moderate income."

The authority operates a number of programs for low-income residents of the county, including public housing projects, rent subsidies, housing construction and construction of new roads and sewers. It also administers numerous loan and grant programs.

Supervisor Thomas M. Davis III (R-Mason), who last week succeeded Scott as chairman of the board's subcommittee on housing, said the plan heralded "a time of more financial accountability and cost effectiveness.

"The board of supervisors is now a full partner in housing and community development policy-making," Davis said. "I think the resources will be there now that the accountability is there."

In the board session yesterday -- the first that has been broadcast live on cable television -- supervisors also received a report that the numbers of women and minorities in the county's work force, though increasing, remained short of county goals.

The report also said that 80 Fairfax County employes complained of discrimination based on age, sex, race, national origin or handicap in fiscal year 1984.