Washington (CNN) -- Three U.S. Secret Service members will leave the agency because of an alleged prostitution scandal in Colombia, the agency said Wednesday.
One of them is a supervisory employee who is being allowed to retire, and another employee has resigned, the agency said.
A third agent, another supervisory employee, is being pushed out, with the agency proposing he be removed. A U.S. official said the agent plans to fight his ouster.
Another eight members allegedly involved in the scandal are on administrative leave with their security clearances suspended.
The employees are accused of bringing prostitutes to a hotel in Colombia ahead of last week's visit by U.S. President Barack Obama, who was there to attend a pan-American summit.
The alleged prostitutes, the youngest of whom are in their early 20s, had all signed in at Cartagena's Hotel El Caribe, where the Secret Service members apparently stayed, flashing their local ID cards. But one of the women, the source said, was involved in a dispute about how much she was allegedly to be paid for the night.
That dispute brought the incident to light and sparked controversy in both countries.
A review board is expected to be created to determine whether the alleged scandal is an isolated incident or emblematic of a broader agency culture, a source said.
"The only way they will prevent this from happening again in the future is to find out if this is one particular case or if it's a pattern," said U.S. Rep. Peter King, R-New York, the chairman of the House Homeland Security Committee.
"We're working and doing our own investigation and whatever we need from the Secret Service we've been getting," King said. "We want a minute-by-minute, hour-by-hour account of what happened, how it happened, what went on, who knew what was happening. And I have no doubt the Secret Service will give us that."
As many as 10 U.S. military personnel from all branches of the armed forces are being questioned about potential misconduct, including five members of America's elite Army Special Forces.
Obama has said he expects a "rigorous" investigation.
Investigators are also looking into whether drugs were involved, according to a separate source with knowledge of the investigation.
Secret Service Director Mark Sullivan has told subordinates to use "all tools available" to conduct the investigation and has made it known that he believes drug testing is within his rights, the source said.
It is not clear whether any of those accused have been tested, and authorities say drug use is not consistent with their findings so far, though they are continuing to investigate the allegations alongside local police.
A U.S. official said the Secret Service interviewed all of the maids at the agents' hotel and the maids said they found no drugs in the rooms.
The scandal is sure to come up when Homeland Security Secretary Janet Napolitano testifies next week at a previously scheduled Senate Judiciary Committee hearing on oversight of the Department of Homeland Security. The hearing is scheduled for 10 a.m. ET Wednesday.
At least one congressman, U.S. Rep. Randy Forbes, R-Virginia, has called for Sullivan to be replaced.
"There's only so many strikes you get, in baseball it's three," said Forbes, a senior member of the House Armed Serves Committee, referencing a 2009 security breach in which a Virginia couple crashed Obama's first White House state dinner, as well as apparent agency overspending in that same year.
"I think he's had three," Forbes added. "I think it's time to put somebody else in there to make sure we're getting a different culture in the Secret Service."
Sullivan has directed the Secret Service since May 2006. He has been with the agency since 1983.
A Colombian official said Wednesday the incident has has overshadowed his country's showcase of the "beautiful, calm city" of Cartagena during the summit.
Nausicrate Perez, a municipal official in Cartagena, said authorities were waiting for more details on the alleged scandal because Cartagena has no confirmation that local people were involved.
Sen. Patrick Leahy, D-Vermont, Senate Judiciary Committee chairman, said he is in close touch with Sullivan and believes he is taking "serious action" to investigate the incident.
The Secret Service agents and officers being investigated range in experience from relative newcomers to nearly 20-year veterans, two government officials with knowledge of the investigation said Monday.
Each agent was offered an opportunity to take a polygraph test, according to a U.S. official. Some of the agents and military personnel maintain they didn't know the women were prostitutes, the official said.
Even so, King said, "it was totally wrong to take a foreign national back to a hotel when the president is about to arrive."
House Oversight Committee Chairman Rep. Darrell Issa, R-California, said he has a high level of confidence in Sullivan.
While soliciting prostitution is in most cases legal for adults in Colombia, it is considered a breach of the Secret Service's conduct code, government sources said. Military law also bars service members from patronizing prostitutes, engaging in conduct unbecoming an officer or, for enlisted personnel, conduct "prejudicial to good order and discipline."
The military personnel allegedly involved were sent to Colombia to support the Secret Service. A military official who spoke on condition of anonymity because of the ongoing investigation told CNN that two of those being questioned are Marines and that Air Force and Navy personnel also are being questioned.
CNN's National Security Contributor Fran Townsend, Juan Carlos Lopez, Dana Bash, Bob Kovach, John King, Barbara Starr, Jessica Yellin, Deirdre Walsh, Ted Barrett and journalist Fernando Ramos contributed to this report.