William Bradford Reynolds, head of the Justice Department's Civil Rights Division, asked a federal judge today to strike down a racial-quota system used to maintain integration in Brooklyn's Starrett City, one of the nation's largest subsidized-housing projects.

Reynolds also said in an interview later that his department is probing use of a similar quota by Montgomery County, Md., to maintain a 50-50 racial balance between whites and minorities in 320 of its 950 public-housing units.

"All I can say is that we have them under investigation," he said.

Reynolds appeared in U.S. District Court in Brooklyn to ask Judge Edward R. Neaher to rule that Starrett City must stop accepting tenants in a way that maintains a population that is about 65 percent white and 35 percent minority, rather than accepting applicants from the top of a waiting list.

About 20,000 people live in Starrett City's 46 buildings, which contain 5,881 apartments. Neaher did not indicate when he would rule.

The project, privately owned but built with state financing and operated with federal and state subsidies, "practices the worst kind of discrimination . . . cloaked in the term 'integration maintenance,' " Reynolds said. The 1968 civil rights act "explicitly forbids" using quotas, he said.

Civil-rights advocates have been divided for years over use of quotas to maintain integration at predetermined levels. Supporters argue that controls are necessary to avoid a "tipping point" at which whites will move out and the housing complex will become "resegregated."

Halting Starrett City's tenant-selection policy "would create a segregated wasteland" in the project, said Morris B. Abram, former vice chairman of the U.S. Commission on Civil Rights and an attorney for Starrett City. Court decisions frequently have approved use of quotas to prevent segregation.

"We admit . . . the process and method selected have to be carefully tailored, and we have done that," Abram said.

Starrett City was first sued in 1979 by the NAACP and two fair-housing organizations on behalf of several blacks on the project's waiting list. The complainants said they had to wait much longer than whites for apartments there.

About 75 percent of families on the waiting list were minorities, but only 30 percent of vacant units were given to nonwhites at that time. About 12,000 people are on the current waiting list, 75 percent of them minorities.

Attorneys reached a settlement in June 1984, leaving the quota system in place but allotting 175 more apartments to blacks and Hispanics over the next five years.

Under the terms, New York City housing officials agreed to set a goal of 20 percent minority occupancy over the next 15 years in 86 other apartment developments.

One day after the settlement, the Justice Department sued Starrett City, alleging that the tenant-selection system violated federal law.

Two weeks ago, the department sued the Redevelopment and Housing Authority in Charlottesville, Va., challenging occupancy controls allowing black families to live in no more than 65 percent of public-housing units and no fewer than 35 percent. NAACP officials there have protested the policy.

At the time, a spokesman said the department planned no action against Montgomery County because it had not received a request to do so.