Until April, acquaintances say that Rep. Frederick W. Richmond (D-N.Y.) had a schedule that began at 8:30 in the morning and never ended until midnight -- full of speeches, fund-raising events, meetings with constituents and special interest groups and elegant parties here and in New York.

Then stories surfaced about a federal grand jury looking into allegations that he had used aides to purchase marijuana and cocaine for his use.

After more than seven years in Congress, Richmond, the wealthiest man in the House, found that he had become a social pariah. Except for routine committee hearings, his colleagues began to steer clear of him. And in May, colleagues asked him to step aside as chairman of the $5,000-a-member Speaker's Club, a fund-raising operation run by the Democratic Congressional Campaign Committee. His presence there had become too embarrassing.

With his resignation from Congress yesterday as part of a federal plea bargaining arrangement, Richmond left behind many critics and a few fans, but mostly people who were baffled by his behavior.

His greatest accomplishments in Congress since his election in 1974 to represent an impoverished section of Brooklyn are seen as his work on the House Agriculture Committee as a champion of food stamps and nutrition programs for the poor and his efforts to promote the arts.

"In the last two years, he was the single most effective and strongest proponent of food stamps in the Congress. He picked up the McGovern banner in January of 1981 . . . as the advocate for poor people. Unfortunately, there is no one who will replace him," said John Kramer, formerly a special counsel to the House Agriculture Committee and more recently a special counsel to Richmond.

Kramer said Richmond's success was not so much in writing and passing legislation as in stopping conservative efforts to cut back food programs for the poor.

Richmond chaired the subcommittee on domestic marketing, consumer relations and nutrition that has fought the Reagan administration as the president sought to push through deep cuts in the food stamp program.

Rep. Leon E. Panetta (D-Calif.), who served with Richmond on the subcommittee, said that Richmond became adept at putting together coalitions of rural and urban interests to preserve nutrition programs.

"In the last couple of years, Fred has had some pretty rough battles. As chairman of that subcommittee, he was the point man on deals to keep food stamp and nutrition programs," Panetta said.

"At first his manner would tend to aggravate the other members whose interests were in the farming area. Food stamps were always the poor stepchild of the Agriculture Committee," Panetta said.

Rep. James M. Shannon (D-Mass.) added, "He was good at it. Fred Richmond could really cut the deals for the liberals on food stamps. He was respected in that area."

Richmond was also an avid supporter of the arts. Besides having a private collection valued at millions of dollars, he also organized the Congressional Arts Caucus last year.

But in spite of his accomplishments in those areas, Richmond is viewed as something of an emigma who will leave behind few close friends in Congress.

Born Nov. 15, 1923, in Mattapan, Mass., a poor, working class neighborhood of Boston, Richmond was an able pianist and formed the Freddie Richmond Swing Band to earn money in college. He earned a B.A. from Boston University in 1945 and set off to earn his fortune in the import-export business.

Rep. Thomas J. Downey (D-N.Y.), a fellow -liberal who was elected with Richmond in the post-Watergate freshman class of 1975, said Richmond had spoken of making a fortune, losing it and going on to earn a second fortune before running for public office.

Besides his business interests, Richmond has been a strong civil rights advocate and served on the New York City Commission on Human Rights from 1962 to 1967. He served on the New York City Council from 1973 until 1975 when he came to Congress.

In Congress, he was popular with the liberals, always ready to throw a fund-raiser or make a speech.

"He was remarkably generous and friendly," said Downey, who was one of Richmond's closest friends in Congress. "He was a very dedicated progressive. It's a great tragedy that his personal life was, to say the least, chaotic."

Richmond also became a powerful force in his Brooklyn district, which is populated by an unusual mixture of upper-income professionals, poor blacks and a wide collection of ethnic groups.

But he alienated some constituents, especially blacks who attempted to run against him, by the large amounts of personal money he he was able to pour into his campaign and into the district.

Much of the money was channeled through the Richmond Foundation, his philanthropic organization, which funneled contributions to churches, YMCAs and other organizations in his district.

Rep. Charles B. Rangel (D-N.Y.) conceded yesterday that Richmond's seemingly inexhaustible supply of money had engendered some resentment in his district from other candidates who felt Richmond had bought his office.

"When you talk about the Richmond campaigns, you can't ignore the money. They say many a church would be foreclosed without the Richmond Foundation. They say there are more organs in the Baptist churches in his district than in any other part of the country," Rangel said.

One colleague described him as a "very confused man" who alternately screamed insults at his staff members and then offered to send their children to college.

He said, "I liked him, but he was an obnoxious son of a bitch. I've seen him abuse staff verbally . . . . He was a bully. He'd have wild, insane moments. He was a very insecure man with 32 million bucks in his pocket.

"He thought he was bigger than the rules," he said.