In the first case of its kind, a federal jury in Alexandria yesterday gave a boost to the U.S. government's legal battle against pornography by finding three Northern Virginia residents guilty of racketeering charges for distributing obscene videotapes and magazines.

The racketeering conviction permits the government to confiscate all assets gained through the defendants' racketeering enterprise.

The case is the first in which federal prosecutors have brought racketeering charges against distributors of allegedly obscene materials, a recommendation made last year by Attorney General Edwin Meese III's Commission on Pornography.

U.S. District Judge T.S. Ellis III is to hear legal arguments today on the forfeiture matter, including whether the government has a right to confiscate materials that have not been declared obscene in court and therefore are protected under First Amendment guarantees of free speech.

Two of the defendants, Dennis E. and Barbara A. Pryba of Lorton, own three area adult book stores and nine video shop outlets, known as Video Rental Centers. The third defendant, Jennifer Williams of Woodbridge, was employed by Educational Books of Upper Marlboro, a Pryba-owned firm.

Assistant U.S. Attorney Lawrence J. Leiser, who prosecuted the case, said the verdict gives distributors of adult materials "some concrete idea of what the community standard is in this district . . . . It will help them assess their own stock and determine whether it's in the limits of the law."

The Prybas and Williams, as well as their attorneys, declined to comment on the verdicts.

The jury of four men and eight women deliberated three days before finding the Prybas, both 45, guilty on three racketeering counts and seven counts of interstate transportation of obscene materials. The Prybas were acquitted of two counts of allegedly filing false income tax returns for 1984 and 1985.

Educational Books was found guilty of two counts of racketeering.

Williams, 38, who worked as a bookkeeper and payroll clerk for Educational Books, was convicted on two racketeering counts and seven counts of interstate transporation of obscene materials. Williams, who is Barbara Pryba's sister, was acquitted of one count of racketeering.

In addition to forfeiture of assets, the Prybas face up to 95 years in prison; Williams faces 75 years.

Dennis Pryba has long been an adult book store retailer in the area and has had several convictions under state laws for selling obscene materials. Educational Books has been convicted 15 times in Fairfax County for violating state obscenity laws and owes the county more than $260,000 in fines from those convictions.

According to evidence at the trial, the Prybas obtained most of their tapes and magazines from a New York-based firm called Model Magazine Inc., described by law enforcement officials as a major distributor of sexually explicit materials.

Officials say Model Magazine is a subject of a grand jury investigation in Alexandria.

Among the items the government wants to seize are Barbara Pryba's Lorton house, valued at $2 million, bank accounts, a warehouse in Upper Marlboro, several vehicles and the videotape and magazine stocks of the Prybas' book and video outlets.

On Monday, the jury is to reconvene to hear evidence from prosecutors and defense attorneys about the couple's finances in order to determine which assets should be forfeited.

At issue in nine days of testimony during the trial were four videotapes and nine magazines seized from the Prybas' shops in raids by local and federal agents in October 1986. Federal prosecutors argued that the materials went beyond what the Northern Virginia community accepts in sexually explicit materials.

The videotapes were played in court and graphically depict heterosexual and homosexual sex, anal sex, sado-masochistic sexual acts and bondage. The magazines contain explicit photographs of similar acts.

Since the couple's indictment in August, racketeering charges have been brought against Reuben Sturman by federal prosecutors in Las Vegas. Sturman was named in the commission's final report as one of the largest distributors of allegedly obscene materials in the country.

The Alexandria case was prosecuted by the Justice Department's newly created National Obscenity Enforcement Unit and the office of Henry E. Hudson, U.S. attorney in Alexandria, who headed Meese's Commission on Pornography.