The U.S. Secret Service on Saturday placed 11 agents on administrative leave as the agency investigates allegations that the men brought prostitutes to their hotel rooms in Cartagena, Colombia, on Wednesday night and that a dispute ensued with one of the women over payment the following morning.

Secret Service Assistant Director Paul S. Morrissey said the agents had violated the service’s “zero-tolerance policy on personal misconduct” during their trip to prepare for President Obama’s arrival at an international summit this weekend.

“We regret any distraction from the Summit of the Americas this situation has caused,” Morrissey said in a statement.

The rapidly unfolding scandal has upstaged Obama’s trip to the summit, where he is discussing trade and the economy with 32 other heads of state. Though the agency has said Obama’s security was not compromised, the allegations of misconduct have brought intense scrutiny to an agency that had not had a major lapse since 2009, when two party crashers entered the White House uninvited.

The situation deteriorated further Saturday when the Defense Department announced that five military personnel, who are staying at the same hotel, violated curfew Wednesday night and have been confined to their rooms. The department will conduct its own investigation upon their return to the United States, said Air Force Gen. Douglas Fraser of the U.S. Southern Command, where the military personnel were from.

Fraser said he was “disappointed by the entire incident and . . . this behavior is not in keeping with the professional standards expected of members of the United States military.”

Rep. Peter T. King (R-N.Y.), chairman of the Homeland Security Committee, said Saturday that Secret Service officials conducting an internal investigation told him that the staff at the Hotel Caribe summoned local police after discovering a woman in the room of one agent after 7 a.m., against the hotel’s policy for visitors of paying guests.

Although the agent eventually paid the woman and she left, King added, police reported the incident to the U.S. Embassy, which informed the Secret Service. The agency quickly recalled the agents and replaced them with a new team before Obama’s arrival Friday afternoon at the Hilton a few blocks away.

King praised the agency for removing the men involved, but he added that “everything they did was a violation of proper conduct.”

“First of all, to be getting involved with prostitutes in a foreign country can leave yourself vulnerable to blackmail and threats,” King said. “To be bringing prostitutes or almost anyone into a security zone when you’re supposed to protect the president is totally wrong.”

Briefing reporters in Cartagena, press secretary Jay Carney said the White House learned of the incident Thursday and Obama was informed Friday.

“This has not been a distraction,” Carney said. “It has been much more so for the press than for the president, who is going on with his work here.”

The Hotel Caribe is in Bocagrande, a seaside district of Cartagena. It’s not a colonial hotel, like those in the old walled city, but rather an elegant, decades-old structure that is considered a national patrimony. Locals consider it a good place to party — there is a beachfront bar-restaurant in front of the hotel and inside it has gardens and bars.

Any presidential trip, but especially those abroad, involve immense manpower and logistical planning that can take place weeks before the president arrives, experts said.

Typically, on a foreign trip, more than 200 federal officials from the Secret Service, Defense Department and White House staff are sent to the site two weeks ahead of the event. Once the president arrives on Air Force One, usually with a support plane and press charter plane in tow, an additional 200 people or more join the original group.

Several people familiar with the Cartagena investigation described a night of partying by members of the advance team, who created enough of a disturbance in the Hotel Caribe that hotel employees asked the group to quiet down more than once.

Prostitution is legal in Colombia, but soliciting women for paid sexual favors is against Secret Service policy. It is not clear how many of the 11 agents, some of whom are reportedly married, had sexual encounters with the women or whether it was clear to all of them that the women expected to be paid.

One person with close ties to the Secret Service, who spoke on the condition of anonymity to speak freely about an ongoing investigation, said he was told by agents that the woman involved in the dispute “freaked out” after she was not paid and banged on walls and doors in the hotel hallways.

But King described a calmer scene. He said that under hotel policy, any overnight guest of a paying guest must leave photo identification at the front desk and leave the hotel by 7 a.m. the next morning.

According to King, one of the 11 women had not left the hotel by 7 a.m. Thursday, prompting hotel officials to knock on the door of the room. When nobody answered, hotel officials summoned police officers, King said.

Once police opened the door, the woman and the agent had a brief dispute over payment, King said, but the agent eventually paid the woman and she left.

Colombian police made no arrests because prostitution is legal in the country, but they turned over to the embassy a list of U.S. personnel staying at the hotel.

King said U.S. Secret Service Special Agent in Charge Paula Reid, based in Miami, rushed to remove the officers from the country Thursday.

Ralph Basham, director of the Secret Service from 2003 to 2006, said he spoke with current agency Director Mark J. Sullivan, and Basham called the agents’ alleged conduct “totally out of bounds.” But Basham defended the agency’s quick action in removing the agents from Cartagena.

“Clearly, they made a huge mistake,” he said. “But to try to tie this somehow to impacting the security of the president of the United States is just outrageous. It did not.”

Staff writers Joe Davidson, Peter Finn and Scott Wilson and correspondent Juan Forero contributed to this report. Wilson and Forero reported from Cartagena.