A Fairfax Circuit Court judge yesterday ordered the owners of a Fairfax County adult book store to take steps to curtail loitering in its store that police allege is frequented by individuals seeking homosexual acts.

Judge Barbara M. Keenan issued the order in response to prosecutors' attempts to shut down Show Place Adult Books on Richmond Highway in Mount Vernon under a public nuisance law previously used to close houses of prostitution and massage parlors in the county. She ordered the store to post uniformed security guards in rear sections of the store by 6 p.m. today, but refused prosecutor's requests to order the store closed.

Prosecutors said they decided to test the law in attempting to curtail activities at the store, which is operated by Crotan Books Inc. and has long been the source of controversy in the county. Yesterday's court hearing followed a two-month police investigation earlier this year in which undercover policemen said they were solicited frequently at the store for sexual favors.

Police investigator Steve Mason testified that on numerous occasions he has been approached by individuals in the store soliciting sexual favors and inviting him to participate in homosexual activities in the projection booths where adult movies are shown.

The store's attorney, Burton Sandler, agreed to posting the guard but said after the hearing that police had exaggerated the incidence of homosexual activities at the store.

Assistant Commonwealth's Attorney Richard F. MacDowell Jr. told the judge that the criminal convictions and subsequent fines levied against the store for selling obscene materials "has not been effective in reducing the kind of behavior that goes on in the Crotan Book Store."

The corporation that owns the store has been convicted of selling obscene materials at least 79 times since 1975, MacDowell told the judge. The most recent convictions came yesterday when a Fairfax jury found the store guilty on seven counts of intent to rent obscene movies and three counts of selling obscene magazines. The jury recommended the maximum $1,000 fine on each charge.

Keenan ordered a July 21 hearing to consider prosecutors' requests to close the store's movie operation housed in the rear of the store. After the hearing, MacDowell said he is hesitant to ask the judge to close the entire operation because of First Amendment protections that cover the sale of magazines and other printed materials in the store.

The store owners are facing pressure also from the state attorney general who has asked the State Corporation Commission to dissolve the store's charter on the grounds that it "has continued to exceed or abuse the authority conferred on it by law."