As Rep. Cardiss Collins, chairperson of the Congressional Black Caucus, looked over the annual showing of black political strength, squirming like students at graduation in the Washington Hilton Ballroom, she listed for the crowd the accomplishments of the caucus' first decade and warned of the tenuous progress of blacks. The Illinois Democrat kept coming back to the refrain: "It is not enough."

By that time Saturday night, as the events began to run three hours behind schedule, the audience of 5,000 was ready to adopt the battle cry as their own slogan of discontent.

In the Very Special Reception earlier that evening, where a smaller group of corporate donors and celebrities had been wooed with egg rolls and spare ribs, one influential civil rights worker bemoaned the waste of a talent pool that comes to town, gladly shells out the costs and then looks for involvement beyond the business of speeches and socializing.

"These people should be directed out to their congressmen. Let them see the voting bloc, and let's show our interest on everything, not just the social justice questions," she said. Though no new sentiments, the frustrations seemed more maddening at this large election-year gathering of black politicians. The Longest Sit

"You mean," said Bill Cosby, his shrill, mocking tone echoing thourghout the ballroom, "they didn't trust us to have food while people were getting wards? They thought we wouldn't look up?" Cosby was kidding, sort of, because he had been to Ben's Chili Bowl within the last 24 hours, but the rest of the hall found the situation humorless. Dinner was not served until midnight, four hours later than expected.

In past years, the awards and speeches and the dinner have been held simultaneously. When the dinner's guest list grew to the thousands and was split between two hotels, those who were not at the hotel where the main speakers -- such as U.S. presidents and stars like Stevie Wonder -- appeared, complained about "secondclass" treatment. This year the caucus tried to seat everyone in one room for the program, which included a show by Gladys Knight and the Pips, and the dinner. Things were slow getting started, and the program didn't wrap up until 11:30. Then the diners were ushered into the exhibition hall of the hotel, where the ceiling duct work and the silver decorations were incompatible, and the hotel was getting ready to leave. By that time most of the guests had grimaces that would have fit into a Grade B movie about prison discontent.

During the program new minority businesses were created on the spot, mainly taking pizza orders and making runs to the candy stand. Eleanor Holms Norton, the chair of the Equal Employment Opportunity Commission, was unabashed, eating one Hershey bar with almonds and stashing another. "The only thing that relieved my anxiety was Gladys Knight," said Norton, at one party afterwards.

The long sit for dinner dominated all the party conversation. "This has been the most disorganized so far," said public relations executive Ofield Dukes. "Now here is why we need a convention center," said D.C. City Council Chairman Arrington Dixon. A sympathetic Shirley Chisholm was hearing all the complaints about hunger headaches and offered, as she sat down to her past-midnight supper, "we just have to tighten up."

Michael Mitchell, one of the Baltimore political clan, may have had the last word. "I think Dick Gregory was the consultant on the dinner." In Absentia

The panel on international relations was high on Janet Diggs' agenda. "One of the areas of focus was Southern Africa, and Charlie was an active champion of the rights of the black majority. I kept thinking he would have been there on the podium right next to Rep. [william] Gray [D-Pa.]," said Diggs. Her husband, the founder of the Congressional Black Caucus 10 years ago, is now serving time in an Alabama federal prison, after being convicted of mail fraud and accepting kickbacks. His name was invoked by Louis Stokes, who said, "Had it not been for the leadership of Charles C. Diggs of Michigan, there would have not been a Congressional Black Caucus."

In accepting a tribute for her husband, Janet Diggs said, "with success, comes enemies . . . when you think we have it made, look around, you better think again." The Real Minority

Would the caucus weekend be better run if Republicans dominated the numbers? Attorney and Reagan worker Samuel Jackson laughed on his way to find his table. "We wouldn't do it differently. After all, making $450,000 is a good idea," he said. "But we would plan it better, manage it better."

Jackson and other Republicans found time to caucus. Attorney George Haley was briefed for his campaign trip next week. Ernest Love, a retired White House employe, conferred with Ronald Reagan recently in Middleburg, Va., where he discussed the use of former labor official Art Fletcher as a point man in the campaign. "That rhetoric that Reagan will put folks back on the plantation just isn't true," Love said. "We said that about Justice [Hugo] Black, and we have to learn to take a chance." Networking

Earlier in the weekend, Antonio Harrison, an Alabama state legislator, was standing in the hall of the Rayburn Building, explaining the personal value of the caucus weekend.

"I can walk into one of these rooms and take care of a whole day of phone calling." Gus Savage, a journalist who has won a primary contest in Chicago and hopes to join the caucus this November, was paying courtesy calls on the House leadership. "I want to hit the ground running, so I won't be so much of a rookie," he said. Judge George W. Crockett Jr. of Detroit, was introducing himself to prospective colleagues.

While the legislative workshop receptions were in full swing, around 30 people met in another room of the Rayburn Building, working out the details of a progressive black political party. Said political scientist Ron Walters, "Some of the caucus members are supportive, some are very cautious. At all the conventions we have missed so many bargaining positions. It makes you wonder whose purpose is being served. We feel we have to have a base underneath the personalities."

Michael Robers, 31, a St. Louis city councilman, was getting some work done. He arrived in town on Thursday, made some calls to contacts at the Department of Energy and the White House on some gasohol projects, skipped the caucus' energy braintrust for three private meetings, met up with D.C. City Councilman John Ray at a brunch to discuss business with the city and then met with a bank president at the dinner. "It works," he said, smiling. Partisan Gaps

The one deliberate tactic of the dinner that worked was the avoidance of partisan speeches. No one said anything against Reagan, for Carter, or about Anderson. This unity was mainly the result of the disunity of the caucus. But the faithful, who are feverishly working for the Carter-Mondale ticket or Reagan-Bush ticket, don't appreciate that silent strategy. "I thought that was interesting but also impolite," said Republican Howard Jenkins.

"It's not the lessor of two evils. The signals coming from the black caucus are very mixed. That's not healthy," said Nate Holden of Los Angeles, at the Carter campaign brunch on Saturday at the Harambee House hotel. "We understand the process now and we can move ahead, away from the poverty program shaping and consulting. If they don't join this campaigning, a lot of people are going to be left behind."

By Saturday morning, Andy Chrishom, of the University of South Carolina, Columbia, had talked to 35 people about their state campaigns. "The work now is to make sure we keep that support intact. And those five minutes conversations tell you how its going." Other Sidelights

. . . Charlene Drew Jarvis, the D.C. City councilwoman, making sure her sister, a new federal appointee, met the right people.

. . . Maurice Dawkins, announcing a countergroup to the conservative Moral Majority. "They will tell you if you elect Carter, you are electing the devil. It is not only the opposite of the truth, it's a lie."

. . . In accepting an award for community service, Sugar Ray Leonard said, "People make people. And all I have to do now is beat Roberto Duran."

. . . Speaking of the caucus dissidence, acting White House chief of staff Jack Watson said quietly. "We do not write them off. In politics you can't have everybody with you all the time."

. . . Gladys Knight pulling Bill Cosby to his feet to sing, then getting an earsplitting ovation with her rendition of "I Will Survive."

. . . The people at M. Carl Holman's brunch, obviously missing the familiar face of Vernon Jordan. Holman repeated the teasing message that Jordan had left on Holman's answering machine that morning: "Be you needing an extra bulter this morning?"