David E. Rivers slid into a chair in his lawyer's downtown office yesterday, looking oddly casual and somewhat haggard in wrinkled clothes and shoes with no socks. No sign here of the well-tailored hauteur of the man who could walk that walk of power only days ago in the District Building.

Rivers, who will be 44 on Saturday, is a man on the spot these days.

His name is in the thick of a wide-ranging federal probe of the District government's contracting procedures. But so far Rivers, a senior adviser to Mayor Marion Barry and the highest-ranking official directly affected by the probe, has not been indicted or charged with anything.

"I feel like I'm fighting ghosts," Rivers said in his first interview since going on leave last week to prepare for any charges that may come. His lawyer, John F. Mercer, sat nearby, deflecting any specific questions about the probe or an FBI search of Rivers' Southwest town house.

They're waiting, Mercer said, for U.S. Attorney Joseph E. diGenova to show his hand. Mercer said he believes Rivers is a target of the probe, but complains that to this point the government seems to have done nothing more than publicly embarrass a proud man.

"Black entrepreneurs in this city and country are very small {in number} anyway," Rivers said in one of his few comments bearing on the investigation. "It's almost impossible not to know a person like Roy Littlejohn {a businessman named in the investigation}."

Rivers suggested that prosecutors unfairly see conspiracy among minority businessmen and public officials when in fact it's all part of traditional business tactics.

"If that's the case, what about all those {restaurants}, what about those {other} businessmen . . . on Connecticut Avenue. What do you think they're talking about?" Rivers asked.

To hear some of his friends and coworkers tell it, the federal investigation is a devastating blow to an ordinary "country boy" who came to the city and worked his way up in the world of public service, first as an Army enlisted man with top-secret clearances, then as an aide for Atlanta Mayor Maynard Jackson in budget and planning in the mid-1970s, then as a mid-level human services official in President Carter's administration, and since 1981, an official for Barry.

As former director of the massive D.C. Department of Human Services, they said, Rivers displayed an arrogant and often abusive style to subordinates who cringed at his outbursts and hated what River admits were his sometimes "scathing" memos. While he was director of Human Services from 1983 until last January, Rivers was embroiled in controversy almost constantly over one or more of the agency's programs. He said the job would make anyone curse and scream.

He is widely credited with bringing under control the department's huge budget, which had been a chronic problem for years.

One friend said Rivers lives the two lives of his astrological sign, Gemini, the twins. "He has two faces," the friend said, intending no insult. "He can scream at me {at work} and two hours later we can go and have champagne and talk about everything but {work} . . . . The person on the job is the image, it's not the David I know."

Rivers, reluctant to analyze his personality, asked a reporter not to call his widower father or other family members, instead suggesting associates in Atlanta and in Washington. He said his modest town house in Southwest was all he could afford.

"The idea that I'm loose and wild and run with a bunch of people . . . is wrong," Rivers said.

Yes, he said, he tries to go to Rio de Janeiro every year for the carnival and hopes to go again next year. Yes, he likes nice clothes and buys them when he can. Yes, he's been divorced since 1968, dates women and has seen his town house become a gathering place of sorts.

At a news conference last week when he stepped down from his post as secretary of the District of Columbia, Rivers tangled with reporters over the issue of alleged illegal drug use. "I am not currently using drugs," Rivers said before dropping the issue. FBI agents reported finding powdery residues believed to be cocaine in Rivers' home.

A native of Berkeley County near Charleston, S.C., Rivers grew up with brothers and sisters in Atlanta. After his stint in the Army, Rivers went to night school at Georgia State University in Atlanta while he worked as a postal worker nearby.

Tapped for an urban fellows program at Yale University, Rivers later earned a master's degree in public administration at Georgia State and worked for the Atlanta Regional Commission, a planning group, before joining Jackson's mayoral staff.

Rivers said he joined the Barry administration when a friend -- he can't remember who -- recommended him to James A. Buford, a former director of the D.C. Human Services Department. He went to work for Buford as a deputy in 1981, and became department head in May 1983.

When Rivers mentions that his federal job in the Carter administration was to help minority businesses get access to the federal government contracting system, his lawyer springs forward with a sarcastic deduction: "That's where the conspiracy started!"