One thing that really rankles Joseph Yeldell is the accusation by critics that in his rise from modest beginnings to power and privilege, he turned his back on the poor.

"It incenses me when people say I forgot. I haven't and never will," Yeldell said. "The one thing people who know me can say is that I'm the same Joe Yeldell who grew up in the 13th Street NW. There has been no change in my attitude or action."

For more than five stormy years Yeldell commanded the Department of Human Resources, the immense, 10,000-employe health and social services agency with a $400 million budget and control over most of the city's programs for improving the lot of the poor.

When he took over in December 1971 the department was troubled by serious problems. The infant mortality and tuberculosis rates for Washington, considered key indices of a city's health, ranked among the worst in the nation. The city's welfare system was constantly being criticized for being insensitive to the needs of the poor, and welfare payments were regarded as being too meager to keep pace with inflation.

When Yeldell left the department in April 1977, the high tuberculosis and infant mortality rates here were largely unchanged even though other cities, including Detroit and New York, had made significant improvements.

Early on, Yeldell sent memos demanding that welfare recipients be treated with respect and he fought successfully to raise welfare payments and Washington began to pay some of the highest benefits of any city in the country. But the rate and Washington, for a time, had the worst record in the nation. The system for enrolling new persons into welfare programs remained so chaotic with backlogs mounting that both Yeldell and Mayor Walter E. Washington were repeatedly sued and were, at one time, cited by a judge as being in contempt of court.

By the time Yeldell left, almost every major agency within the sprawling department had been criticized, investigated or threatened with the loss of federal funds.

"What I faced at DHR was an almost unbelievable combination of factors - bickering, entrenchment by bureaucrats, people running around with titles, but no fuction," Yeldell said in a recent interview. Placing the Blame

"When I came and took those titles away, people said outright: I'll fix him. Well I decided to take the weight. I wanted to show that I would be the director, the only director. It worked for a while. Then, the people got sick. Then every DHR problem became Joe Yeldell's problems."

That they became Yeldell's problems had much to do with his style: Yeldell refused to delegate authority, refused to allow subordinates to answer questions from the press and make decisions. If he wanted to take all credit for mastering the agency, in the end he took all the blame for his failure.

In one of the many reorganizations of DHR, in 1975, he said he would shift his role to become, he said, the "principal advocate" for the city's needy and give authority to four commissioners to run the department.

"We're giving them the responsibility, we're giving them the authority and we're going to be holding them accountable," Yeldell said then. It was one of the rare public instances in which he spoke of the city's poor with a clear feeling of concern. Most of his public utterances were fixed in the bureaucratic jargon of the welfare technocrat.

He thought he could run a bureaucracy with the words of bureaucrats: management by objective and zero based budgeting became catchwords that replaced, according to his critics, effective action to improve the lot of impoverished patients in overcrowded wards at D.C. General Hospital, the welfare applicant waiting months for benefits to begin, or the children who once would have gone to the city's orphanage who lived for months and years in hospitals and mental institutions because DHR had failed to find other homes for them.

"What made me upset," said Polly Shackleton (D-Ward 3) whose council committee oversees DHR operations "was that time and time again we kept finding instances of people not getting the kinds of services DHR was supposed to be providing."

"(Yeldell) was a macho man," said William Barr, head of the social rehabilitation administration in DHR. "The social scientists in the department said he had an authoritarian personality."

"Many people began to see him building a political base with tremendous resources at his disposal," Barr said. "I remember hearing people talk. They said if they didn't knock Joe off soon, he was going to be dangerous. The idea was that anyone who got DHR working like they wanted it to would have a good thing going politically." Assessments by Others

"I think he was really trying to make DHR work," Barr said. "he was concerned that all the blame was hanging around him, but the time he tried to do something about that, delegate more authority, the die had been cast."

Said another of Yeldell's associates, who asked not to be identified, "He couldn't see what was really happening to him. He thought he was doing just fine."

"DHR had always had problems," City Council member Marion Barry said. "I knew that. But this man didn't have the skills to manage it, so it just got worse."

Today, Barr said, management of DHR programs is "basically the same as when he left."

But, he said, "The attack on the department has softened, which allows us to do some of our work. The department and Joe became one and the same. Whether you went to the council, to the Justice Department, you never knew whether they were attacking the department or Joe Yeldell. The funniest thing I've ever seen in my career was to see the person and the department become one and the same."

He referred to life with Yeldell as responding every day to a crisis, with employes stopping the flow of their work to defend themselves against the latest allegation of mismanagement of a program for example. "We all got tarred with the brush of incompetent DHR management. You're managing a mess with all the attacks." Tigh-Money Problems

Yeldell claimed that many of the problems he faced as DHR director could be resolved if he had more money in his budget. But he became DHR director at a time of inflation, with spiralling Medicaid costs, congressionally mandated increased salaries for DHR employes, increasing costs for constructing and maintaining hospitals and other institutions, and a rising cost of living that was pushing up the cost of basic welfare assistance programs. It was not a period of congressional largess for anything beyond the most necessary programs. New social service programs for helping retarded children in institutions, for example, were quickly cut from the budgets.

As problems within the agency became more severe, Yeldell held steadfast. Many on Capitol Hill, in the City Council and DHR itself argued that the agency was simply too big, too combersome for any one person to manage. But both Yeldell and the mayor fought tenaciously against any move to dismember it.

At budget hearings before the City Council, Yeldell maintained that a superagency to coordinate the various health and social services programs for the city's poor was the best way of delivering services and that, given money, staff and time, he said, he could make it work.

But time ran out. As he wound up his fifth year on the job in late 1976 and early 1977, newspaper stories alleging nepotism, cronyism and leasing abuse in DHR led key advisers to Mayor Washington to tell him that Yeldell was becoming a serious "political liability." Emergency in New Role

But, months later, Yeldell reemerged as a key political adviser to the mayor and his general assistant, the number three official in the city government after the mayor and city administrator Julian Dugas.

The mayor gave Yeldell responsiblity for helping run a $85 million program to give city contracts to minority firms. Nearly a year after the program was created, none of the money had been allocated and Yeldell again was in the midst of controversy.

Early this month, at a City Council hearing, Yeldell charged that it was the council's failure to provide budgetary resources that had caused problems. But, Milton G. Carey, former vice chairman of the commission created to award the contracts, apparently referring to Yeldell, said "It is not the law that has not worked; it is the people who have not worked."

The mayor also had Yeldell help set up a program to ease the problem of high unemployment among teenagers here. Yeldell helped design a $30 million prgram with federal money for 3,200 youths who would spend nine months in training, graduating with skills to be city government parking meter readers and water testers. City officials who supervise such jobs said they doubted that many positions would be available for graduates. The program has not yet been started.

Recently, reflecting on the criticism that he had forgotten the poor of Washington, Yeldell said he thought such statements might have reflected his move from Anacostia to a home in North Portal Estates in the fashionable neighborhood off 16th Street known as the "Platinum Coast."

"Just because we were looking for a larger home and found a good buy in North Portal, we weren't trying to isolate ourselves," Yeldell said. "My daughter still spends as much time in Southeast as she does at home. People in my family are still poor - and there is not a person that I have met that I won't stop and talk to, whenever or wherever I see them."

In the coming months, Yeldell will have to defend himself against an indictment, alleging that when he ran DHR and helped arrange a city lease of a building owned by real estate millionaire Dominic F. Antonelli Jr., he committed the crimes of bribery and conspiracy and "defrauded the citizens of the District of Columbia."