A Massachusetts public housing agency that is facing a lawsuit by the Justice Department has agreed to stop selecting tenants according to a racial quota system designed to maintain integration in its four apartment projects.

The Holyoke Housing Authority had used the tenant selection system since 1977 as a way to keep the racial composition of the buildings' 694 units at half white and half minority residents. Under the system, whites on the waiting list were given apartments as often as needed to maintain the 50-50 balance, even when it meant skipping over minority families who had been waiting longer.

A lawsuit charging the Holyoke agency with violating the 1968 Fair Housing Act and the 1964 Civil Rights Act by giving preference to whites over Hispanic and other minority applicants was filed last week in federal court in Massachusetts. But a consent decree, settling the suit and reflecting an agreement between the Holyoke agency and the Justice Department, was filed at the same time. The decree requires the housing authority not only to stop using the quotas but also to find and offer apartments to Hispanics who were passed over in favor of whites.

Holyoke's quota selection plan ironically was the result of a 1973 agreement with the Department of Housing and Urban Development under which the Holyoke agency was committed to integrate its public housing. HUD, which said Holyoke officials were violating the 1964 Civil Rights Act forbidding discrimination in federally funded programs, either would have stepped in and run the housing authority itself or cut off federal money if the housing authority had not integrated the buildings, according to Raymond Murphy Jr., executive director of the Holyoke agency.

HUD approved use of the quotas at the time of its agreement with Holyoke, Murphy said. When the Justice Department threatened the Massachusetts agency with a lawsuit, Holyoke officials responded that the only way to comply with the civil rights act was by violating the 1968 Fair Housing Act forbidding discrimination in housing on the basis of race, according to Murphy.

Although many advocates of fair housing disagree, the Justice Department interprets use of a quota in selecting public housing tenants as a violation of the 1968 legislation.

Several major court decisions have held that quotas do not violate the Fair Housing Act, according to Oscar Newman, a consultant on housing integration plans and community planning. "My real concern is that the Justice Department, in giving emphasis only to the nondiscriminatory intent of Title 8 of the Fair Housing Act and not to the intent to integrate, is effectively facilitating all assisted housing becoming all-minority," he said.

Holyoke and the Justice Department, with help from HUD and state officials, now have hit upon a plan that should result in integrated public housing without use of quotas, Murphy said.

The Holyoke authority will give its public housing units to whites on its list and help Hispanic families find housing in the private market, using financial help from HUD and the state, to the extent necessary to keep the racial balance in the housing projects. When each family comes up on the list, it will receive either a public housing apartment or a rental assistance certificate, according to Murphy.

HUD took part in negotiating the consent and "fully supports it as rectifying a practice which our Boston regional office originally found to be in violation" of the 1964 Civil Rights Act, said John J. Knapp, the department's general counsel.

If successful, the Holyoke agreement "could become a model" for public housing authorities in other areas with similar problems in maintaining integration, Murphy said.

Public housing agencies in Montgomery County and Charlottesville, Va., use quota systems to maintain integration in their housing, although Montgomery's policy affects only about 320 of its 950 units. HUD, which must approve tenant selection plans in housing receiving federal funds, has delayed decisions on the Montgomery County and Charlottesville systems.

The Justice Department, on the other hand, has sued Starrett City, a large New York City housing project that uses a tenant selection system based on racial quotas.

The Holyoke public housing agency attracted the disapproval of the Ford administration because three of its apartment buildings were inhabited predominantly by whites, while nearly all the residents of the fourth were minorities, Murphy said. Most minority tenants in Holyoke's public housing are Hispanics, who make up 14 percent of the city's population of 43,000. Blacks comprise 2 percent.

After the 1973 agreement, the three buildings predominantly housing white tenants were integrated with the use of the quotas. The housing authority never was able to integrate the fourth, which predominately housed Hispanics, however, because "white families wouldn't move in," Murphy said. Of families on the waiting list for public housing in 1973, 70 percent were white and 30 percent were Hispanic. By 1982, the proportions had reversed, with nearly 90 percent of the applicants Hispanic and 10 percent white, according to Murphy.

As a result, more than 500 Hispanic families suffered when "recent white applicants were chosen and many qualified Hispanics who have been on waiting lists for years were bypassed," according to the Justice Department lawsuit.

As part of the settlement, the housing agency agreed to "initiate immeditate efforts" to find families who were skipped over on the waiting list and to offer public housing units to them in the chronological order of their original applications. The agency may not evict or transfer tenants to comply with the order, however, if the residents are occupying "appropriately sized units."

The Holyoke housing officials also were ordered to tell the minority families affected by the court decree that they can use Section 8 rental assistance certificates in private housing instead of moving into public units, and to help them find apartments if they choose the private market.

The certificates allow low-income families to move into private apartment buildings where landlords participate in the program, paying 30 percent of their income for rent. HUD then pays the difference between the 30 percent of family income and the fair market rent established by the federal agency. The Department of Housing and Urban Development has given Holyoke 50 Section 8 certificates and the state has given 15 similar certificates for use by families covered by the court decision.