The White House yesterday put distance between itself and Thomas B. Ellis, President Reagan's nominee to the Board for International Broadcasting, and a majority of the Senate Foreign Relations Committee wrote Reagan asking that the nomination be withdrawn.

"The record of Mr. Ellis, as well as his testimony, will impair, in our view, his effectiveness on the board," said the letter, signed by the committee's eight Democrats and Sen. Charles McC. Mathias Jr. (R-Md.).

There was no immediate comment from the White House.

Ellis, 62, admitted in his confirmation hearings before the panel on Tuesday that he once directed a group that funded research into theories that blacks are genetically inferior to whites. He also acknowledged having extensive holdings in South Africa and having recently visited that country at the expense of its apartheid government.

Ellis, a major fund-raiser for Sen. Jesse Helms (R-N.C.) and manager of Reagan's victorious 1976 North Carolina primary campaign, also said he strongly opposed school desegregation in this country in the 1950s. However, he told the committee, "I do not believe in my heart that I'm a racist."

White House spokesman Larry Speakes said yesterday that he did not know if any administration official had been aware of Ellis' background before the nomination. Speakes added that Reagan had "repudiated" an Ellis-run effort in the 1976 North Carolina primary to circulate fliers on Reagan's behalf claiming that President Ford favored a black running mate.

Speakes told reporters at the White House that Reagan was not aware until he read yesterday's newspapers that Ellis was involved in that incident.

"Obviously, the president does not subscribe to, or condone, racist views," Speakes said.

Joining Mathias in sending the letter to the White House were Sens. Claiborne Pell (D-R.I.), Joseph R. Biden Jr. (D-Del.), John Glenn (D-Ohio), Paul S. Sarbanes (D-Md.), Edward Zorinsky (D-Neb.), Paul E. Tsongas (D-Mass.), Alan Cranston (D-Calif.) and Christopher J. Dodd (D-Conn.).

Earlier in the day, White House officials were said to be reluctant to withdraw the nomination on the chance that Ellis could repair the damage with added testimony before the committee. However, no further hearings were scheduled.

"The president has made a great to-do about his sensitivity on racial issues," Tsongas said Tuesday in calling for the nomination to be withdrawn. "It's time to see if he is really serious . . . . Given the president's recent attempt to portray the administration as sensitive on racial matters, this nomination is hypocritical in the extreme."

In another development, the White House announced yesterday that Reagan had overruled his staff and will speak at a Veterans of Foreign Wars convention Aug. 15 in New Orleans.