"I have a very interesting relationship with the Black Caucus. Sometimes we have been in harmony, sometimes I am not in complete agreement," Jimmy Carter told 3,200 paying guests at the Congressional Black Caucus (CBC) dinner on Saturday. "In low times my letters from Parren Mitchell are addressed 'occupant.'"

They roared. It wasn't supposed to be that way. This was to be a 'fess up moment between the President and the year's largest assembly of black leaders. It was the time the President was supposed to give something to a constituency that feels it has had little return on the power it demonstrated at the voting booth last November. Instead of policy they got puns. And the black audience loved it, for the moment.

For the past three days, Capitol Hill, the Washington Hilton Hotel and many private homes, have buzzed with the hour-to-hour activities of the CBS weekend. This gathering has grown over the last seven years from a one-night, black-tie dinner to three days of political meetings, parties and the hybrid reception that's both.

It is a place to be seen, no matter what your age or politics. In public on Friday evening, Andrew Young, the U.S. ambassador to the United Nations, called the weekend the "Mascot Nigger Convent." What he meant was that it is an assembly of those who have made it in politics, the arts, business and other public arenas.

The faces were mostly black. But Sens. John Glenn and George McGovern and Rep. Newton Steers, joined some of the functions.

Besides the 16 officials who make up the CBC, Lt. Gov. George Brown of Colorado. Rev. James McIlwaine, a councilman in East Spencer, N.C., and Mayors Richard Hatcher and A. J. Cooper represented politics. The faces of the 1960s - organizers Jesse Jackson and Ralph Abernathy, widows Coretta King and Betty Shabazz - mingled with the ascending leadership. Amelia Parker, Robert Malsom and Valerie Pinson of the White House, Ernest Green of Labor and Haskell Ward of State. Add a few traditional stars - Arthur Ashe, Ruby Dee, Jerry Butler, Lou Gossett, Larry Brown and Dick Gregory - and an unduplicated happening occurs.

Besides the appearance of the President to give an extra dimension this year, the weekend drew other contacts. Ford Motor Co. flew 13 executives down from Dearborn, Mich., in the company jet. It is a spin-off and other groups, such as the Minority Architects and Contractors, plan their annual meetings and fund-raisers around it.

It is a once-a-year captive audience, producing its own tense, funny, poignant and ordinary moments.

"You probably noticed I was a little late," said Jimmy Carter, as he walked onto the stage of the Hilton ballroom with his wife, Rosalynn. "But I met ("Roots" author) Alex Haley in the hall and made the mistake of saying 'how's your family?' and unfortunately, he told me."

All the amimosity dissolved, the same crowd that minutes earlier had cheered Parren J. Mitchell, the chairman of the Caucus, for saying, "It is immoral, it is immoral, to keep people out of work . . . We cannot permit the attacks on affirmative action to continue without launching our counterattack," roared at the President's diverting one-liners. And Carter was in top form, anxious not to bomb as he had at the National Urban League convention in July, days after a stinging attack by its director, Vernon Jordan.

Standing in front of Mitchell, Haley and Young, he spoke of economic justice, saying he hoped that one day "we all" could afford Rep. Ronald Dellums' tailor.

When times were going bad for the administration, Carter continued, he could count on Young to divert the media. "Unfortunately he taught Bert Lance a few things so I guess Andy and I are alone again this week." As he left, after 20 minutes, Carter waved and said, "I love you." Not madly, as Duke Ellington used to say, but I love you, nonetheless.

As the dinner broke up, and the serious partying began, the consensus was that the President hadn't said anything. Asked if she agreed, Rep. Shirley Chisholm demurred, calling it a pleasant, social evening. Vernon Jordan smiled and said, "This isn't my meeting." Ossie Davis, the emcec for the evening who said of Carter on stage, "I hadn't seen that many teeth since Louie Armstrong died," later commented, "I was profoundly disappointed that his coming was the whole of it." Even Jesse Jackson was gracious, noting, "We have been so alienated from the presidency and the Presidents that we get such gratification from him showing up. I'm glad he did, but it didn't cancel out the need for substance."

The weekend is now a media event. ABC and David Wolper Productions were filming every move of Alex Haley for a documentary to be aired on the first birthday of "Roots" next January. WHUR-FM, the Howard University radio station, broadcast live from the dinner. "Harambee," Channel 9's black-oriented public-service program, was doing a special, filming from the legislative workshops on Friday morning to the wee hours on Saturday night. Dozens of press people received credentials.

Naturally, a lot of pushing occurred. At one point, a film crew, all white, pushed ahead of Maurice Sorrell, the pint-size photographer for Jet and Ebony, to get near Young and Haley. "We're the working press," one of them screamed, Sorrell turned around and yelled back, "We were working with these guys before you discovered them."

A moment the camera didn't capture.

Late Friday evening, about 75 friends of Rep. Cardiss Collins (D-Ill.) who chaired the $125-a-plate dinner on Saturday, gave her a surprise birthday party. About midnight Coretta Scott King arrived, the women embraced and Mrs. King said, "I wish my sister here a happy birthday. We do have a lot in common, I admire her strength." As they talked privately, everyone watched the two women, one's husband a symbol snatched away by an assassin, the other's husband, a congressman killed in a plane crash. They embraced again and Mrs. King led a vigorous chorus of "Happy Birthday."

Three times during the weekend people bowed their heads in moments of silence for Steve Biko, the South African black leader found dead in his prison cell two weeks ago. Parren Mitchell asked, "If, if indeed the children of Soweto dare to act against naked racism, dare we do less against the American-style racism?"

South Africa vied with South Newark for attention over the weekend. Richard Hatcher of Gary, Ind., a mayor with a typical urban problems, demonstrated this dual concern as he sponsored a fund-raiser for Trans-Africa, a new lobbying group.

"In a sense the linkages between the domestic conditions of minorities and the liberation movements in Africa are more obvious than ever before," said Hatcher. By January, the organizers hope to raise $100,000 and start operations. "When a foreign aid bill comes up, the Jewish lobby gets mobilized immediately. We don't have that strength," said Herschelle Challenor, staff consultant for the House Subcommittee on Africa. "We have to fight for $50 million for the Sahel but someone can say this is wrong in India and they get billions. We've got to balance that out."

When a group of diplomats and economists from Zaire arrived at the Washington Hilton on Saturday, they couldn't get rooms. So the delegation, eight men and one woman, sat near the lobby talking in rapid French, spreading out papers across the cocktail table, oblivious to the photographers taking pictures of them. Kasongo Mutuale, the ambassador-designate, said, "We don't mind having to go elsewhere because of the Black Caucus, and it's kind of nice being confused with the activities here."

"Hey, girl, guess who I'm going to the dinner with," Delores Handy, anchorwoman and reporter for Channel 7, called out to a friend in the Hilton lobby on Saturday morning. "Lou Gossett called, he knew me from television in Los Angeles, and asked me to go to the dinner." Handy's friend acknowledged that Gossett, who won an Emmy for his portrayal of the fiddler in "Roots," was certainly the catch of the weekend, not only for his talent but for his looks. "He can rattle my chains anytime," the friend confided, and on Saturday evening Handy was beaming.

The essence of the CBC weekend is found, perhaps, in the assembly of 475 black women at 8 a.m. on Saturday morning, singing a cappella a song about Fannie Lou Hamer, the Mississippi civil rights activist who died earlier this year, and listening to her gravelly voice lament in a film, "I'm sick and tired of being sick and tired."" When Fannie Lou Hamer's mighty face disappeared from the screen, there was silence.

The salute to Mrs. Hamer was part of a breakfast and workshop sponsored by the National Hook-Up of Black Women, Inc., a group that developed out of the weekend two years ago. "There were just too many women here for a good time. We wanted to use those talents," said Arnita Young Boswell of Chicago, the national chairwoman. Part of the breakfast menu was 250 pounds of sausage patties, bacon and smoked links, purchased from the Parker House sausage firm, a black-owned company in Chicago.

Other Scenes . . . Alex Haley mobbed at Morgan's Crab House on Georgia Avenue by just plain folks and receiving the same adulation from former Sen. Hugh Scott and Secretary of the Army Clifford Alexander at a brunch . . . EEOC Commissioner Eleanor Norton doing a mean boogie at the after-dinner party given by CBS Records, who donated $12,500 to the Caucus . . . Solicitor General Wade McCree sitting, unsmiling, as President Carter joked about the "gracious" assistance the Caucus gave the Justice Department in preparing the brief for the Bakke case and later saying he hadn't heard the comment . . . Dorothy Height, the head of the National Council of Negro Women, leaving Walter Washington's party at 2 a.m., moving to another . . . and Jimmy Carter looking very humble as he said "Alex Haley and I both wrote books last year. Mine was called 'Why Not the Best.' His was."