The Congressional Black Caucus has blocked the appointment of a white man, Marshall Smith, as acting commissioner of education by threatening to oppose President Carter's proposed education department, sources said yesterday.

Members of the 17-member caucus, led by Rep. Parren J. Mitchell (D-Md.), reportedly told the White House and aides to Vice President Mondale that Smith was unacceptable because a book he helped research had "racist overtones."

Caucus members reportedly told the White House later that if Smith received the appointment, they would vote against the education department bill when it comes up for a final vote in the House next month. The vote on the bill, one of Carter's last big government reorganization initiatives, is expected to be close.

A White House source said some Black Caucus members said they would try to have the group take a formal unified position against the bill.

Fearing the opposition of the caucus to Smith, White House and Mondale aides then pressed Smith to withdraw and Undersecretary to Health, Eduction and Welfare Hale Champion to nullify the appointment. But Champion, arguing that Smith was not a racist, strongly resisted.

Smith, however, finally concluded that he should step down to avert conflict and withdrew his name Monday afternoon in a phone conversation with Champion. He will remain as assistant education commissioner for policy studies. He had been chosen for the acting commissioner slot about a week ago by HEW Secretary Joseph A. Califano Jr., effective July 1.

Mary Berry, a black and assistant secretary of HEW for education, was named yesterday to fill the post of acting commissioner for 30 days.

Friends of Smith in the educational community scoffed at charges he may have racist ideas.

"He is no more a racist than the man in the moon," said David Breneman, a senior fellow at the Brookings Institution. "It's absolutely outrageous.

One White House official, a leader of those who urged both Smith and Champion to withdraw the appointment, declared yesterday that he does not believe Smith has racist ideas.

However, he said, after members of the Black Caucus had warned they had doubts about Smith and might vote in a block against the embattled education department bill, he urged Smith to withdraw.

Smith was one of the principal researchers on the 1972 Christopher Jencks book "Inequality," although the conclusions of the book are attributed solely to Jencks. That book held that "equalizing educational opportunity would do very little to make adults more equal," that "the long-term effects of segregation on individual students are quite small" and that "the characteristics of a school output depend largely on a single input - namely, the characteristics of the entering children."

Jencks said family background was the most important factor.

It was these conclusions that came under attack from Black Caucus members, although Jencks also concluded that "blacks . . . can and should be guaranteed the right to attend predominantly white schools."

Mitchell and Champion could not be reached for comment.

The commissioner's job became open because of the resignation of Ernest Boyer. It would have been an interim appointment until the new education department is approved by Congress.