Cartagena, Colombia (CNN) -- President Barack Obama called Sunday for a "thorough" and "rigorous" investigation into allegations involving prostitutes and Secret Service agents in Colombia.
Some 11 Secret Service agents and officers are being investigated over preliminary findings that they allegedly brought back several prostitutes to a hotel in Cartagena, U.S. government sources familiar with the investigation have told CNN.
"What happened here in Colombia is being investigated by the director of the Secret Service," said Obama, who spoke in Cartagena, where he was in town for the Summit of the Americas event.
"I expect that investigation to be thorough and I expect it to be rigorous. If it turns out that some of the allegations that have been made in the press are confirmed, then of course I'll be angry," he said.
The alleged misconduct occurred before Obama arrived in Cartagena. The Secret Service personnel have since been sent back to the United States and put on administrative leave, the agency said. The U.S. military said that five U.S. troops who were working with the Secret Service are also under investigation for missing curfew and alleged "misconduct" at the same Colombian hotel.
"We're representing the people of the United States and when we travel to another county I expect us to observe the highest standards," said Obama. "Obviously what's been reported doesn't match up with those standards."
Still, he cautioned: "I'll wait until the full investigation is completed until I pass final judgment."
None of the agents or officers being investigated was part of the president's personal protective detail and Obama was not based at the Hotel Caribe, where the alleged misconduct occurred. But dignitaries and journalists reporting on the hemispheric meeting were staying there, a U.S. government official said.
Rep. Peter King, chairman of the House Homeland Security Committee, was briefed on the matter and told CNN on Saturday that the government personnel brought prostitutes back to their rooms Wednesday night and "one of the women did not leave the room in the morning."
A hotel manager tried to get in the room, and eventually the woman emerged and said "they owed her money," according to King. Similarly, U.S. government sources said there was a dispute between at least one Secret Service member and a woman brought back to his hotel over a request to be paid.
At least one of the women brought to the hotel talked with police, and complaints were filed with the U.S. Embassy, the sources said.
"The Secret Service saw that report, and they immediately began an investigation," King said.
Secret Service spokesman Edwin Donovon said that the agents were relieved of duty Thursday -- prior to the president's arrival in Colombia -- and replaced after "allegations of misconduct."
The agency's assistant director, Paul Morrissey, noted his agency's "zero tolerance policy on personal misconduct."
"This incident is not reflective of the behavior of our personnel as they travel every day throughout the country and the world performing their duties in a dedicated, professional manner," Morrissey said Saturday. "We regret any distraction from the Summit of the Americas this situation has caused."
The U.S. military is conducting its own investigation and will mete out "punishment, if appropriate ... in accordance with established policies and the Uniform Code of Military Justice," U.S. Southern Command said in a statement Saturday.
Ronald Kessler, a former Washington Post reporter who has written a book about the Secret Service, called the incident "clearly the biggest scandal in Secret Service history."
The Washington Post, which was the first to report the story, said it was alerted to the investigation by Kessler.
Jon Adler -- president of the Federal Law Enforcement Officers Association, which represents Secret Service agents and other federal law enforcement officers -- urged caution in jumping to conclusions, characterizing the incident as "isolated" and not necessarily a scandal.
"That's just sort of an overdramatic interpretation of an isolated incident," he said. "We have to trust the process of the internal review."
While soliciting prostitution is legal in certain areas of Colombia, it is considered a breach of the agency's conduct code, the government sources said. High-level officials in the Secret Service and the Department of Homeland Security were outraged over the incident, the sources said, noting that the investigation indicated the prostitutes were brought back to a hotel that had been secured for the summit.