Alan L. Keyes, the highest-ranking black in the State Department, has submitted his resignation after charging that he was treated in a racist manner by Deputy Secretary John C. Whitehead.

Keyes is a political appointee who had been serving as assistant secretary for international organization affairs, the office that deals with the United Nations. His resignation was tendered after Whitehead, at a meeting Friday, overruled Keyes in a dispute about the distribution of U.S. financial contributions to U.N. agencies. The sources said Keyes complained afterward that Whitehead had talked past him to his subordinates in a manner that he regarded as a racial snub.

"There is just no truth whatsoever to the allegation that this had a racial aspect," Whitehead said last night. "It's true we disagreed over what I consider a rather minor administrative issue, and I think Alan is making a mountain out of a molehill. He has done a good job, and we regret to see him go but he has made his decision."

Keyes declined to discuss the incident except to say his resignation will become effective Nov. 13. Department spokesman Charles E. Redman said Secretary of State George P. Shultz had accepted the resignation "with regret." Redman added that he would leave it to Keyes to give the reasons for his departure. Keyes said he intends to accept a research and writing appointment at the American Enterprise Institute.

Keyes, formerly a member of the career Foreign Service, has a reputation as an articulate and outspoken neoconservative who strongly supports President Reagan's foreign and domestic policies. He is regarded as a protege of former U.N. ambassador Jeane J. Kirkpatrick and resigned from the Foreign Service in 1983 to work for her at the United Nations. He was named to the assistant secretary post last year.

In addition to his duties involving the U.N., Keyes sought repeatedly but unsuccessfully to have Shultz designate him as the principal spokesman for the Reagan administration's controversial policy on South Africa. However, despite Keyes' strong backing for the administration's opposition to economic sanctions and disinvestment by American firms, Shultz resisted giving Keyes a role that would have impinged on the prominence or authority of Chester A. Crocker, assistant secretary for African affairs.

Department sources said that Keyes frequently was on the losing side of internal policy disputes. The sources said the reason appeared to be because his extreme conservatism frequently put him to the right of the more centrist policies favored by Shultz and other senior department officials.

The sources said that Keyes reacted in an increasingly frustrated manner, complaining that he felt like a "token black" whose opinions were not respected. The sources said he had threatened on at least one occasion to resign over his inability to win a bigger role in articulating Africa policy.

Whitehead, a former Wall Street investment banker, has a reputation for informality and courteousness, but also is known for standing firm once he has made a decision. Several sources said they had never seen him exhibit any hint of racist attitudes, and some pointed out that he has been one of the department's strongest advocates of greater efforts to provide opportunities for minorities in the Foreign Service.

Keyes was understood to have received a letter yesterday from Shultz expressing his "regrets" and describing Keyes as a valuable member of the State Department team.