Robert S. Hoffman of the University of Kansas, one of the country's leading experts on mammals, has been selected as new director of the Smithsonian's National Museum of Natural History.

The appointment, announced yesterday by Smithsonian Secretary Robert McC. Adams, marked the first time that the prestigious job has gone to a scientist from outside the Smithsonian. The museum employs 120 scientists and curators.

Smithsonian Undersecretary Dean Anderson said the choice was made on grounds of "demonstrated leadership, not just on scholarly ability." Anderson explained that at the University of Kansas, where Hoffman has been a professor since 1968, Hoffman had served as the "moral equivalent of the dean" in temporary tenure over the School of Arts and Sciences, "a job in which he served not just adequately but near to brilliantly."

Anderson said that he anticipated "no substantial change of policy" under Hoffman. "There is no dissatisfaction factor in this decision," he explained. "No right degree turn, no left degree turn."

Hoffman succeeds Richard S. Fiske, a geologist who was director from January 1980 to July 1985. Fiske is returning to the museum's Department of Mineral Sciences to resume his research on volcanoes.

In a telephone interview yesterday from the university at Lawrence, Kan., Hoffman, who is 56, said he and the Smithsonian had agreed to an initial term of five years. "After that," he added, "it is customary to sit down and take stock, and examine how things have gone, and where we want to go from here." As to the present state of the museum, Hoffman observed that "the programs are excellent now. And I'm not coming in to fix anything."

One of the conditions of Hoffman's acceptance was that he be able to continue his research in systematics and ecology. "It deals," he said, "with the evolution and ecology of living mammals. The work may shed additional light on how evolution works. If we develop an understanding of how a species evolves, we then may come to know how societies evolve."

As an example, he described the field work he did in Canada's Yukon and Northwest Territory last summer. "We were looking for a particular kind of shrew, to examine its chromosomal structure," said Hoffman. If the animal had chromosomes like those of the Siberian shrew, it would lend weight to theories on passage over the Bering Strait. If it resembled those in eastern Canada the conclusions would support differing theories. Did they find the shrew? "Well," replied Hoffman with a chuckle, "you don't think I'd be telling you about it if we failed." It resembled the Siberian animal, by the way.

Heads of the natural history museum are chosen by an unorthodox system that begins with classified ads in newspapers and magazines. In this case, a selection committee deemed 33 on their preliminary list to be qualified. The field was narrowed to three, who were asked to lead symposiums at the Smithsonian as part of the selection process. Hoffman said he was asked to take the job in mid-June, but that his final decision was delayed by his research in the far north and by negotiations on the terms. "For instance," he explained, "I had to work out things like how my research would be supported. In the past I got a lot of support from the National Science Foundation, for which I would no longer be eligible."

Prior to going to Kansas, Hoffman served 13 years at the University of Montana. He speaks and reads Russian. He was a member of the U.S.-U.S.S.R. Joint Commission on Science Policy of the National Academy of Sciences from 1974 to 1982 and was on the NAS Advisory Committee on the U.S.S.R. and Eastern Europe from 1970 to 1975.