Thomas Ellis, President Reagan's nominee to the Board for International Broadcasting, yesterday asked that his name be withdrawn in order to stop "ultraliberal senators . . . and their ilk" from using him to frustrate the president's economic and defense policies.

In a letter of reply, Reagan said he regretted the circumstances that led Ellis to withdraw, and said Ellis behaved throughout the affair "in a manner consistent with the reputation for honor that you have always possessed."

Meanwhile, the president defended his own record on civil rights, telling 300 women attending a White House reception for the National Council of Negro Women yesterday that he felt self-conscious addressing them because of reports that "I am prejudiced, if not an outright bigot."

The Ellis nomination ran into trouble Tuesday when Ellis acknowledged in testimony before the Senate Judiciary Committee that, in the 1976 Republican presidential primary in North Carolina, he distributed fliers with an openly racist appeal that accused President Ford of considering a black running mate if he won the nomination. Ellis was Reagan's North Carolina campaign manager.

In addition, Ellis acknowledged that he has large holdings in South Africa and recently was a guest of that country's apartheid government. He also said that he was the director of a group that funded William Shockley's research on whether blacks are genetically inferior to whites.

Nonetheless, Ellis told the committee he did not believe that he is a racist.

In his letter to the president yesterday he said the hearings had ceased to be a discussion of broadcasting "U.S. views behind the Iron Curtain" and had become a "personal attack," led by Sens. Joseph R. Biden Jr. (D-Del.) and Paul E. Tsongas (D-Mass.).

"It was an obvious partisan political effort to drive a wedge between you and the black community using me as the instrumentality . . . ," Ellis said. "Frankly, I do not mind taking whatever heat these two ultraliberal senators want to generate in my direction. However, I believe it is vital to America that they and their ilk be denied the opportunity to use me to hinder your struggle to return economic stability and provide an adequate national defense to our nation."

Reagan did not withdraw the nomination, according to White House officials, because he wanted to avoid angering Sen. Jesse Helms (R-N.C.) and prevent criticism that he was "running scared" on the race issue.

Ellis has been a key fund-raiser for Helms, the chairman of the Judiciary Committee, which previously recommended rejection of two other administration nominees: arms control director Kenneth L. Adelman, who subsequently was confirmed by the Senate, and Ernest W. Lefever, who subsequently withdrew his nomination as assistant secretary of state for human rights.

After Ellis' letter was released, Tsongas, who with eight other committee members wrote the White House asking that the nomination be withdrawn, said Reagan was "caught between Jesse, and his need to be consistent with his recent statements on racial sensitivity."

"Any nominee before this committee is responsible for what he's done in the past," Tsongas said in response to Ellis' claim that Democrats had turned the hearings into a personal attack. "His views on racial matters are relevant."

Meanwhile, the president told the National Council of Negro Women that he has been frustrated by the "totally false image that has been created of me" regarding race.

"I'm more than a little self-conscious facing you here and saying these things," Reagan said in a slow, solemn voice. "There has been such a case made that I am prejudiced if not an outright bigot that I find myself wondering if you might not be thinking that I don't mean what I'm saying, that this is just political hot air. Believe me, it is not."

Reagan appeared self-conscious in reading the speech.

He told the women that his father had prohibited him from viewing the "movie classic, 'Birth of a Nation.' " Realizing that he had called the racially insensitive movie a "classic," the president explained in halting words that he regards the film as a classic not for its content but for charting "a new course in movie-making."

At another point, the crowd groaned when the president said his administration has proposed "the longest extension of the life of the Civil Rights Commission in its history, 20 years . . . ."

Reagan has fired three members of the commission, and critics have accused him of "stacking the panel" with replacements who oppose quotas and busing.

As evidence of his efforts on behalf of civil rights, the president cited his actions proposing a fair housing law with "real teeth" and filing suit against Alabama over segregated colleges and universities.

"This isn't a grandstanding stunt because next year is an election year," he said.