James G. Banks, 84, a housing and community affairs expert who became an innovative and controversial anti-poverty leader in Washington in the 1950s and 1960s, died June 11 at Casey House hospice in Rockville. He had idiopathic pulmonary fibrosis.

When the city's urban renewal program began, Mr. Banks was selected as the director of family relocation and housing rehabilitation for the D.C. Redevelopment Land Agency, a program to clear Southwest Washington's slums and replace them with upper-income apartments and townhouses. He helped relocate 25,000 people and 1,400 businesses.

Although the program was deeply resented in some quarters -- many in the black community labeled it "Negro removal" -- Robert C. Weaver, a former secretary of the U.S. Department of Housing and Urban Development, said "it was the first time a good job of relocation had been done. It was handled humanely and professionally."

In 1955, the D.C. League of Women Voters named Mr. Banks citizen of the year for his relocation efforts. In retrospect, he later told The Washington Post, he had "ambivalent feelings" about that work.

In 1963, he began a stormy four-year tenure as the first executive director of the United Planning Organization, the District's War on Poverty effort. Under his direction, the organization grew in two years from one person with a $35,000 budget to 1,100 employees and a $21 million budget.

When the War on Poverty was a new concept, he was the architect of several reform-minded programs, many designed to involve the poor in the management of their affairs.

He also proposed turning some public schools over to universities to run them, to see if they could do better than an entrenched school administration. He suggested an experiment in giving birth control devices to unmarried teenagers who had "background characteristics indicating a probability of pregnancy." Churches were not pleased.

The soft-spoken, pipe-smoking Mr. Banks believed in government assistance to the poor and also preached personal responsibility. He often found himself at odds with civil rights groups, members of Congress, public housing advocates and others who believed he should have been more forceful and outspoken on behalf of the District's impoverished residents. Activists in those organizations sometimes viewed him as cautious and plodding.

Washington Post columnist William Raspberry, writing in 1998 about the legacy of self-help among black Washingtonians, described Mr. Banks as "wise, experienced and personally involved in community building in the city where he was born 77 years ago -- to parents who also were born here." Civic success, Mr. Banks told Raspberry, depended on the determination of people to do for themselves and to integrate their efforts into the development of the city as a whole.

James Gouverneau Banks was born in the Barry Farm section of Anacostia, where he was a member of the Epworth League, a Methodist youth group that required debate on current issues and regular prayer. He took part in the discussions despite a severe stutter that he overcame through his own verbal exercise regimen. At his eighth-grade graduation ceremony at Birney Elementary School, he flawlessly recited the class poem, Henry Wadsworth Longfellow's "Excelsior."

He graduated from Dunbar High School in 1936 at age 15. His motto, according to the school yearbook, was "keep a'plugging away."

He was a pre-med student his first year at Howard University and then switched to sociology, studying under the tutelage of sociologist E. Franklin Frazier. After graduation, he received a master's degree in sociology from the University of Pittsburgh.

Following Navy service during World War II, he became assistant director of tenant selection for the National Capital Housing Authority and gradually moved into positions of greater responsibility.

Leaving the United Planning Organization, he worked from 1967 to 1969 at the federal Department of Housing and Urban Development, where he was credited with implementing a policy of citizen participation in urban renewal. He then joined D.C. Mayor Walter E. Washington's Cabinet as director of housing programs.

He ran unsuccessfully against then-Del. Walter E. Fauntroy in 1974, became executive vice president of the Washington Board of Realtors and held various teaching, board and consulting positions.

In 1988, he helped found the Anacostia/Congress Heights Partnership for the Prevention of Homelessness, which highlighted his belief in grass-roots efforts and individual initiative.

"It's awkward to talk about," he once said, "but those community leaders, the citizens association leadership of pre-home-rule Washington, tended to be well-educated, often college-trained people whose leadership grew out of their personal achievement. They were not just role models, though they were that, but they also made the neighborhoods work even for the impoverished residents. We need to bring them back."

At St. Mary's Episcopal Church in Washington, he was senior warden and chairman of the church's restoration committee. He also was a member of the Brookland Literary and Hunting Club and the Cosmos Club.

His first marriage, to Jessie Bray Banks, ended in divorce. Three sons from that marriage died: David Banks in 1975 and Franklin Banks and James Banks Jr., both in 1990.

Survivors include his wife of 32 years, Peggy Banks of Washington; a son from the second marriage, Peter Banks of Washington; a brother, Quentin Banks of Washington; three granddaughters; and seven great-grandchildren.

Mr. Banks and his son Peter wrote a book, "The Unintended Consequences: Family and Community, the Victims of Isolated Poverty," published last year.

James G. Banks helped lead D.C. housing and anti-poverty agencies in the 1950s and 1960s.