Nicoh Mayhew was about to be the state’s star witness in a double-murder trial when he visited his brother in the Prince George’s County jail last fall.

The two met in a narrow cinder-block visiting booth, separated by a wall with a window, a metal vent and a coat of faded teal paint. As they began to talk, the door behind Mayhew’s brother swung open. Two familiar faces — the men that Mayhew had fingered in the killings — crowded inside.

It was the kind of face-to-face encounter between witness and accused that is supposed to wait until court. It was not supposed to happen at the jail, and certainly not to Mayhew, who had been offered a spot in a witness protection program to remain shielded from the two murder suspects.

One of them — a nephew of Mayhew’s — leaned toward the glass and asked his uncle if he had “decided” what he would do. It was a clear reference, prosecutors allege in new court filings, to the testimony Mayhew could offer about the night in 2011 when he said he was called to a remote site and asked to help cover up the killing of two men.

In December 2012, weeks after the encounter in the jail and as the trial for his nephew neared, Mayhew, 25, was executed — ambushed and shot repeatedly in the face as he carried his 2-year-old son up a flight of stairs to his mother’s apartment.

More Post coverage of the Maryland jail scandal.

Prosecutors have charged the nephew with murder in Mayhew’s death, alleging that he orchestrated a hit from behind bars.

But the case has also sparked administrative and criminal investigations that have revealed security flaws at the Prince George’s jail in Upper Marlboro, according to court records and several law enforcement officials with knowledge of the inquiries. They spoke on condition of anonymity because the case against Mayhew’s nephew has yet to go to trial, and investigative work into jail operations is ongoing, they said.

The security issues at the Prince George’s jail are coming to light after federal prosecutors alleged widespread collusion between guards and violent gang members at a state-run detention center in Baltimore.

In the Prince George’s jail, another of the busiest jails in Maryland, administrators have little information about inmates’ contact with the outside world. Unlike at most jails in the D.C. area, Prince George’s does not directly monitor or record visits with friends or family, and inmates routinely shield their calls from investigators monitoring recorded phone lines.

Yolanda Evans, a spokeswoman for the Prince George’s Corrections Department, said that jail officials have cooperated with police investigating Mayhew’s killing, but that she is unaware of any ongoing probe of jail operations.

Evans acknowledged that the episode in the visiting booth should not have happened, saying an internal investigation revealed that a corrections officer who had been on the job just two months mistakenly violated jail protocol. She declined to say whether the officer had been reprimanded, saying the issue was a personnel matter.

Before and after that encounter, however, there were other problems at the jail, court records and interviews indicate. Jail administrators for some time failed to separate Nicoh Mayhew’s nephew, Brian Mayhew, 21, from his co-defendant in the double slaying. As Brian Mayhew allegedly plotted his uncle’s killing, he was able to talk daily with Kenan Myers, 26, in a common area, according to two law enforcement sources with knowledge of the investigation.

Brian Mayhew’s attorney did not return calls seeking comment.

Jail operators also allowed Myers to share a cell with Nicoh Mayhew’s brother, who was awaiting trial in an unrelated case, one of the officials said. That might have made it possible for Myers to learn that Nicoh Mayhew planned to visit, the official said.

Evans said the jail depends on prosecutors to file court orders when co-defendants or family members should not be boarded together. She said there was no direction to keep Brian Mayhew and Myers separated until this spring, when prosecutors asked that Mayhew be moved to another county.

Evans acknowledged, however, that the jail conducts interviews with incoming inmates to assess family connections and other affiliations as officials decide where to house people.

The investigations also revealed a pattern of deception by inmates to foil monitoring of their recorded phone calls, officials said. Prince George’s inmates make calls using debit cards with distinct authorization codes that allow detectives to track calls of particular defendants. But in Prince George’s, inmates routinely swap cards.

When Brian Mayhew allegedly told hit men where and when to find his uncle in December, he used another inmate’s authorization code, law enforcement officials said. It took a series of covert efforts, including asking inmates to wear wires, to unravel how those calls were made, the officials said.

The investigations have also found instances of county corrections officers smuggling cellphones to inmates, according to a law enforcement official with knowledge of the investigation.

The fallout from the Mayhew case has raised questions about the thoroughness of security improvements promised by Prince George’s officials after a 2008 incident in which an inmate accused of killing a police officer was found asphyxiated in his cell.

A year-long investigation followed, but it never became clear whether Ronnie L. White, 19, who authorities said had run down the officer with a pickup, was killed or took his own life, a federal judge said at a recent hearing. One guard was convicted of obstruction of justice. But because 175 jail cameras were then incapable of recording any video, it was impossible to review what occurred around White’s cell.

After White’s death, the jail nearly doubled — to 345 — the number of cameras, and each now records video that is saved for at least seven days.

But the Mayhew case revealed that the jail’s surveillance system still has at least one major blind spot. There is no audio or video surveillance of visiting booths, which are nearly soundproof to corrections officers stationed outside.

Most other jails in the region have the technology to record conversations between inmates and visitors. In the District, family and friends are directed to a room with 54 desktops with video monitors, and inmates have access to similar terminals. Each video-conference is recorded. In neighboring Montgomery County, inmates and visitors speak through phones that can be recorded.

Despite the visiting-booth encounter between the Mayhews, Evans said the jail has no plans to install cameras or to otherwise update its visiting procedures. She also said that tightening procedures for phone calls is not a priority and that officials are spending limited funding on other improvements, such as a new kitchen for the jail, which opened in 1985 and houses 1,300 inmates.

Seeking to prosecute the murder cases with no recording of the alleged intimidation in the visiting booth, Prince George’s assistant state’s attorneys Christine Murphy and Jason Knight recently asked the court to allow them to play Nicoh Mayhew’s grand jury testimony at trial.

That testimony, they said, would explain why Brian Mayhew would want his uncle dead. Nicoh Mayhew was the only witness to his nephew’s alleged involvement in the 2011 double killing, and if convicted, Brian Mayhew could have faced life in prison.

Brian Mayhew’s trial for the 2011 killings had been scheduled for this week, but the merging of that case with the new charges of witness intimidation and the killing of Nicoh Mayhew pushed it into next year.

According to court filings, Nicoh Mayhew had told the grand jury that on the night of May 30, 2011, he was at a family cookout when he received a call to help Brian Mayhew.

Nicoh Mayhew was asked to buy gasoline and a bottle of bleach and meet his nephew near a quarry in Capitol Heights. As he arrived, he later testified, he heard gunshots and saw a car containing two bodies.

His nephew and Myers allegedly loaded boxes from one of the victim’s cars into Nicoh Mayhew’s car, and doused the car holding the bodies with gasoline. They stopped short of igniting it, however, because neither had a match or lighter.

Months later, and not long after the jailhouse encounter with his nephew, Nicoh Mayhew told family members that he feared for his life.

On the day before Nicoh Mayhew was killed, Brian Mayhew called two convicted bank robbers and told them where and when to find his uncle, according to police. Brian Mayhew spoke in code, police said. The time period is “9-11,” and look for a “white girl named Kia,” he said, according to a recording of a jail phone call.

Nicoh Mayhew’s girlfriend drove a white Kia on the day of his killing, which occurred at 9:54 a.m.

In a call later that day, court papers and officials say, one of the alleged gunmen laughed when he reported back to Brian Mayhew.

“Your man lost his mind,” he said. “Your man lost his mind all over the place.”

Dan Morse contributed to this report.