Work on the Whitman Park housing development began here today after 24 years of bitter controversy.

The 120 townhouses long planned for the white, working-class neighborhood in South Philadelphia became a symbol to the scheme's opponents of the conflict between neighborhood rights and government interference in the lives of working people.

To supporters of the $10 million federalaly financed project, construction of the units was an important civil rights issue.

Laborers from the A&R Development Corp. of Baltimore began erecting a fence around the six-acre site in a morning drizzle.

The construction start was no cause for celebration, said John A. Bowser, executive director of the Philadelphia Urban Coalition, a civic group that fought for the housing.

"Today is the day that indicates there is a heck of a lot of work to be done to reverse the racial animosity that exists," he said. "The real work of people living together . . . is still before us." Police officers in yellow slickers guarded the weed-covered lot as disgruntled neighbors watched from their stoops. There were no demonstrations, though Whitman Park has seen many in the past decade.

In 1971, the Whitman Council, a neighborhood organization formed to fight the development, successfully prevented construction when then-Mayor Frank L. Rizzo supported their cause.

His opposition was racially motivated, U.S. District Court Judge Raymond J. Broderick ruled in 1976. Today the judge ruled that half the housing population must be black and half white when the construction job, expected to take 18 months, is completed.

The Department of Housing and Urban Development cited the former mayor's opposition when it withheld more than $20 million in community development funds from the city last May. Rizzo's stand also contributed to Philadelphia being ruled out of consideration for HUD's Urban Development Action Grant program for distressed areas.

The Whitman fight began in 1956 with community protests against what was then proposed as a high-rise project for the site. Court battles and protests changed the nature of the project to a townhouse plan and held up construction until now.