Fred C. Davison, president of the University of Georgia, conferred with White House officials yesterday as a candidate to succeed T.H. Bell as secretary of Education.

Davison told reporters that his first round of interviews left him with the impression that selection of a Bell successor is "not moving as fast as the general public perceives it is moving."

John R. Silber, president of Boston University, and William J. Bennett, chairman of the National Endowment for the Humanities, have been regarded as the leading contenders for the Cabinet post.

Davison said he has "no way of judging" his chances for succeeding Bell, who is quitting Dec. 31 to return to Utah.

Sen. Mack Mattingly (R-Ga.) recommended Davison for the post two weeks ago after Davison was passed over for the chancellorship of the Georgia university system. Davis has been a leading opponent of Title IX of the Education Amendments of 1977, which prohibits sex discrimination in educational programs that receive federal funds.

"I need to assure everyone I'm not really looking for a job," Davison said.

But, he added, "I think when you have concerns and have been a fairly outspoken critic of some of the things going on, if folks offer you a chance . . . to do something about it, you need to do something about it or quit talking about it. That's kind of the position I'm in."