A federal jury said it was deadlocked tonight in the case of a Navy physicist accused of using the Internet to solicit sex from a minor, leading to a mistrial and a rare setback for an FBI unit that targets suspected sexual predators through online stings.

The jury of nine men and three women said it was unable to reach a decision after 11 hours of deliberations. The physicist, George P. Chambers, was the first person to contest charges, brought by the Maryland-based FBI unit in U.S. District Court here, subjecting himself to often sordid testimony about sexually explicit e-mails. Since 1995, more than 50 defendants have pleaded guilty to charges in similar cases brought by the unit.

The jury reported that it was split 9 to 3 in favor of conviction. Judge Andre M. Davis then declared a mistrial.

Chambers, 45, of La Plata became the target of an FBI sting after striking up an e-mail correspondence with an undercover agent posing as a 13-year-old cheerleader. He was arrested June 6 at the Mall in Columbia, where prosecutors said he intended to rendezvous with his e-mail partner for sex. He denied any wrongdoing.

The case hinged upon two issues: whether Chambers believed he was corresponding with a 13-year-old and whether he intended to have sex with her.

In testimony Wednesday, Chambers told the jury that he was simply acting out a fantasy in cyberspace and that he thought all along the purported cheerleader was an adult or even a member of law enforcement who was role-playing.

Chambers, a married father of two preschool daughters, is on unpaid leave from his job as a weapons designer at the Naval Surface Warfare Center. He declined comment last night, and prosecutors declined to say whether they will retry him.

The three-day trial provided a public look at the methods of Operation Innocent Images, an FBI initiative launched in 1995 to thwart people who engage in online child pornography or child sex exploitation. Agents pose as teenagers in online chat rooms, where graphic conversations take place, in hopes of identifying sexual predators.

Chambers and his attorney, Bryan A. Levitt, focused on Chambers's state of mind.

On the witness stand, Chambers acknowledged that he sent the sexually explicit messages and photographs that prosecutors introduced as evidence. Attempting to explain this conduct, Levitt told the jury in closing arguments that Chambers was a "nice guy with a crazy twist" who used the Internet to fulfill fantasies. Chambers testified that he went to the mall out of curiosity, with no intention of having a sexual encounter.

Keith Floyd, a juror who voted for conviction, said he was disappointed that "a bad person" remains on the street.

But Sarah Shifflet, the jury forewoman, said she voted for acquittal because "I don't necessarily believe that he intended to meet this person, whoever it was, for sex."

The FBI's Baltimore office launched Operation Innocent Images as an offshoot of the investigation of the 1993 disappearance of George "Junior" Burdynski, a 10-year-old Prince George's County boy, who has not been found. That case led authorities to two pedophiles who sent electronic messages soliciting sex from boys. That prompted the FBI to take a much closer look at activities on the Internet.

The operation has expanded to 23 other FBI offices nationwide, with the Baltimore office maintaining a central file of information gathered across the nation. Roughly 2,200 people have been convicted nationwide in the probes.

The number of investigations launched nationwide has grown from 113 in the program's first year of operation to 2,370 in fiscal 2002.

"The problem is worsening -- it's not getting better," said Peter F. Brust, assistant special agent in charge of the FBI field office in Baltimore.

In the Baltimore-Washington area, an FBI task force has about two dozen agents and analysts based at an FBI branch in Calverton, using an office building off Interstate 95. The team also conducts training sessions for law enforcement authorities stationed around the world -- teaching them, among many things, how to sound like a teenager online.

The agents work in a secluded room filled with cubicles and computers, posing as teenage boys or girls, waiting to be approached by adults interested in passing on child pornography or meeting for sex. To avoid concerns about entrapment, they are trained not to take the first step. But they go along with the sexual chatter.

The Chambers case followed a typical investigative pattern. At her workstation in Calverton, agent Emily Vacher was in America Online's "I Love Older Men" chat room shortly after 4 p.m. Feb. 1, posing as a 13-year-old cheerleader from Columbia.

In short time, Vacher testified in court, Chambers sent her an instant message and wrote: "are you a cheerleader?" and "do you like older guys." He sent a 10-year-old photo of himself and graphic photos.

Over the next four months, he talked of getting together. He also pushed to talk to her by phone, but she declined. On March 24, he asked whether she worked for the FBI. She replied: "since when does the FBI hire teenagers?" He asked a second time that day if she worked for any law enforcement agency. "Uummmmm no," she replied.

Vacher sent him a photograph April 10: a picture of herself at age 13. Chambers responded, "you are really hot looking."

On May 21, Chambers again voiced suspicions, asking, "Are you a police officer or do you work for any law enforcement agency?"

"No," she responded. "I'm in the 9th grade."

"good, thankyou . . . lots of cops online pretending to be girls."

Vacher typed back: "they don't let 9th graders in the CIA!!!!!"

During the trial, Levitt did not challenge the legality of Vacher going undercover. Authorities explained that FBI agents and police can misrepresent themselves during undercover investigations on the street or in cyberspace.

The June 6 meeting was set up outside a bakery on the mall's second floor.

Investigators then followed Chambers to the mall from his workplace at the Naval Surface Warfare Center in Indian Head, a 60-mile drive, and arrested him outside the bakeshop. Chambers contended that he was walking away when he was apprehended and told the jury that he could not believe he could be arrested for walking at a shopping mall. But the prosecution said he appeared to be milling around, waiting.