In what he described as just the beginning of a shake-up of revitalize a broadly criticized District of Columbia housing department, new housing director Robert L. Moore yesterday removed five of the department's top administrators from their positions and ordered them to report to the city's personnel department for new jobs.

Moore, suggesting that other reassignments will follow, noted: "This process is not over. It's just starting. There may be others, wherever there are those who are incompetent.... I hold my top administrators accountable to me, just as the mayor is holding me accountable to him."

The five administrators are Monteria Ivey, who headed the public housing division; Roy Priest, chief of policy, planning and evaluation; Carroll Swanson, chief of neighborhood improvement; Reginald Webb, executive manager of the department, and John Stone, chief of administration and management. They form a majority of the top rank of Moore's staff.

Moore said the deputy administrators of the various divisions will serve at least temporarily as acting chiefs. But he added that "we may be looking for new talent.... We expect some new people." He said the recruiting process had not begun, however.

Moore said the five former administrators have been told to report to the personnel department this morning for their new assignments.

Personnel director George Harrod told a reporter yesterday evening, "At this juncture, I don't know what assignments I will give them... I will utilize them as best I can." Harrod said four of them are GS-15s, with the fifth perhaps a GS-16. They will keep their salaries, he said, which could range from $38.000 to $47,590 for a GS-15.

Mayor Marion Barry had made a sharp improvement in the city's housing performance one of his most urgent pledges during his campaign last year. He promised that one of the first things he would do if elected would be to remove housing director Lorenzo W. Jacobs Jr. from his post, which he did, as well as others who, he said, lacked "competence and compassion."

The former housing administrators join a long line of employe holdovers from former Mayor Walter Washington's administration who have been reassigned to other jobs. Such administrators have civil service protection and cannot be fired outright.

Moore said yesterday that he is "leaving all options open" about the future of his administrators. Some "adverse actions" -- firing the employes through regular civil service procedures -- may be taken against some of them, he said. Others may end up in other positions with the city or federal government, while others may retire, he said. Some may even come back to the housing department, he noted. Those decisions will be made within the next 120 days, he added.

Last week, Moore spoke to 1,000 employes of the department at a meeting. He told them he held them responsible for their conduct and performance. He was also quoted as saying, "I don't fire clerks and maintenance people. I fire directors."

Some of those removed yesterday are veterans of the city's housing department. Ivey has a long record with the department and National Capital Housing Authority. and Swanson has been in housing-related positions with the city government for more than 25 years.

Priest said he has been with city housing divisions for 11 years and has been head of policy, planning and evaluation since 1975. Priest's only comment yesterday about his removal from the position was: "I'll report over there to personnel tomorrow morning."

Ivey had no comment on his removal, and the other three could not be contacted.