Quiet anger set the tone of the Congressional Black Caucus' 15th annual legislative weekend.

In three days of panel discussions and congressional hearings, in a prayer breakfast yesterday morning and in the hotel hallways before the Congressional Black Caucus' annual awards dinner last night, talk was at a low boil.

There was angry talk of South Africa. South African Bishop Desmond Tutu, a scheduled speaker, was unable to attend. Attempts by caucus members and others to further toughen U.S. policy toward the white minority government are at a standstill.

The caucus chairman, Rep. Mickey Leland (D-Tex.), spoke of "more racism in the country." And House Budget Committee Chairman William H. Gray III (D-Pa.) spoke of President Reagan's "attacks on affirmative action."

But gone were the loud denunciations of Reagan and the effects of his domestic spending cuts now that the president is in his second term despite a lack of black support.

Gone, too, was last year's talk of the controversial split between members of the caucus supporting Walter F. Mondale and those supporting the Rev. Jesse L. Jackson for the Democratic presidential nomination. Jackson appeared to be a welcome face at the prayer breakfast, where he received a standing ovation and heard cries of "Jesse, Jesse."

From the sold-out fashion shows to the exhibit area where companies gave away souvenirs, there were words of pride in the success of the 20-member black caucus, five of whom head full committees in the House. There is particular pride in Gray's handling of the Budget Committee. "The caucus is now a power inside Washington because of who its members are, not because it is a grass-roots organization that can mobilize voters," said Tom Cavanaugh, a specialist in black politics at the National Research Council.

"They have done a superb job as a collective group standing for liberal policy in what is not the most salubrious climate for liberals," said M. Carl Holman of the Urban Coalition.

"The caucus has grown in purpose," Jackson said. "They represent a national black political institution, a forum around which federal policy for black people revolve . . . ."

Gray is not shy in singing the praises of the caucus: " . . . It is the group in Congress for the last four years who has consistently proposed an alternative budget, a balanced budget . . . who raised the tax equity and fairness issue . . . . It was Ron Dellums D-Calif. who began to look at the MX missile."

"The popular issue today is apartheid," the Rev. Joseph Lowery of the Southern Christian Leadership Conference said after the prayer breakfast.

"The reason it is so popular is it reminds us of the apartheid remaining here in our country," he said. "The caucus can see it first hand in the Congress, where the priority is on military spending not life support, not ending poverty or hunger. We see American apartheid in unemployment . . . . There is an expanding black middle class, but there is an even bigger expansion in the black under class."

Internal disputes among the nation's black leaders over issues continue to bubble among the leaders, however. Some of them surfaced at the prayer breakfast during an emotional speech by former Washington Redskins' chaplain Tom Skinner, who said he is disturbed when black politicians believe "our progress is being stopped by one person in the White House . . . the climate in Congress . . . a shift in the mood of the Senate . . . . We are disproportionately preoccupied with the enemy."

Skinner reminded the caucus that a quarterback doesn't call timeout to tell the coach that he could make more touchdowns if "those 11 guys on defense would get off the field."