A federal appeals court has agreed that a Virginia law designed to shield youngsters from obscene magazines is too broad and has said that booksellers who challenged the law may recover their legal costs from officials in the state.

The 4th U.S. Circuit Court of Appeals declared in a ruling released yesterday that District Judge Richard L. Williams acted properly when he declared the 1985 law unconstitutionally broad in September.

The law's "most serious flaw . . . is its breadth," the three-judge panel agreed, noting that booksellers would have to restrict adults' access to books and materials to comply with the statute. The statute, sponsored by a Northern Virginia legislator, banned the display of materials that, even if not obscene, could be considered "harmful to juveniles."

The appeals court reversed a portion of Williams' decision that barred a coalition of booksellers and publishers from receiving reimbursement for their costs in overturning the law. An appeals court panel declared that police chiefs Charles T. Strobel of Alexandria and William K. Stover of Arlington and the State of Virginia could be held liable for those costs.

"We do not find it unjust that the taxpayers will have to bear the costs," the court ruled, because ". . . citizens of Virginia will likewise continue to enjoy unfettered freedom of expression" as a result of the ruling.

"I didn't pass the law and I've not enforced the law," Stover said. "So I don't see the rationale or logic for the court's decision" that the county must pay lawyers' fees, he said.

Bert Rohrer, a spokesman for state Attorney General Mary Sue Terry, whose staff defended the law, said she will probably appeal the ruling to the Supreme Court. "We feel we have a strong case; we feel it's a good law," Rohrer said.

In a suit filed in Alexandria last July, the coalition of local and national booksellers and publishers argued that the Virginia law violated their First Amendment rights and would require them to remove many classics and best sellers from their shelves, creating "adults only" sections or refusing juveniles entry to their stores.

The measure, sponsored by state Del. Vincent F. Callahan Jr. (R-Fairfax) was intended to bar newsstands from displaying such sexually explicit magazines as Playboy, Penthouse and Hustler. It was an amendment to a 1975 statute barring the sale of sexually explicit material to juveniles, a provision that is not affected by the court rulings.

"Forcing a bookseller to create a separate, monitored adults-only section . . . or taking the materials off display and keeping them 'under the counter,' unreasonably interferes with the booksellers' right to sell the restricted materials and the adults' ability to buy them," the appeals panel said.

"In sum, we feel that the amendment discourages the exercise of First Amendment rights in a real and substantial fashion," it said.