Washington (CNN) -- "Worthy of trust and confidence" is the motto of the almost 150-year-old U.S. Secret Service, and Director Mark Sullivan now faces the dual task of proving it true and keeping his job.
A scandal involving allegations of partying with prostitutes in Colombia shortly before President Barack Obama's arrival last week for the Summit of the Americas has forced a rare spotlight on the internal workings of an agency labeled as secret.
Sullivan, a Secret Service veteran for almost three decades who was sworn in as director on May 31, 2006, is the focus of demands by the White House and Congress -- and a perplexed American public -- to figure out exactly what happened.
At issue is how 11 agents ended up in a potentially compromising situation in violation of agency standards. More important are questions raised about whether it was a lone incident or indicative of a culture far removed from the legendary discipline and integrity exuded by the agency.
So far, three Secret Service members are leaving over the scandal -- one forced out, one retiring and one resigning -- and more resignations are expected.
While one member of Congress has called for Sullivan's ouster, other influential legislators and officials quickly defended his initial steps in dealing with the scandal.
White House spokesman Jay Carney told reporters Thursday that Obama has "faith in the director (and) confidence in his leadership," adding: "We are not going to prejudge outcomes and discuss the future of this agency in a press briefing while this investigation is going on."
"The fact of the matter is this is an incident that requires investigation," Carney said. "The Secret Service has acted with speed in addressing the matter, investigating the matter, holding people accountable, and continuing to push forward with the investigation."
Senate Judiciary Committee Chairman Pat Leahy, D-Vermont, said he has been in close touch with Sullivan and believes the agency director is taking "serious action" to investigate the incident, while House Oversight Committee Chairman Rep. Darrell Issa, R-California, said he has a high level of confidence in Sullivan.
Steps taken so far include bringing home the agents involved, stripping them of their security clearance and questioning that has included at least one polygraph exam. In addition, sources said Sullivan wants to form an outside panel to examine if the Cartagena incident reflected a problem of wrongdoing embedded in agency culture.
"He's outraged by this," CNN National Security Contributor Fran Townsend said of Sullivan after talking to him.
Rather than making Sullivan the target or fall guy, members of Congress want to give him the opportunity to investigate it and bring in outsiders to look at the agency culture, Townsend said Thursday.
Despite their initial praise, legislators made clear they need more answers. Issa and the ranking Democrat on the oversight committee, Rep. Elijah Cummings of Maryland, sent Sullivan a letter demanding comprehensive information on the incident.
Cummings told CNN of a conversation he had with Sullivan on Wednesday -- before the departures of the initial three Secret Service agents were announced -- that explained the culture of pride permeating the Secret Service.
"I could tell that he was very upset, and probably more upset than a lot of the people on (in Congress) can be," Cummings said Thursday. "... I asked him a question. I said, 'Look, I get the impression that there's a culture of pride, of excellence, of being the best.' I said, "Is it possible that these men might resign?' He said he had absolutely no doubt that they would."
Cummings continued: "They have this pride they don't want any bad apples and so it probably would be so uncomfortable to them that they would leave. So, yesterday's actions with regard to folks leaving and being fired did not surprise me one bit."
To Congress and the nation, the Secret Service symbolizes the highest level of protective security.
Most visible are the agents who guard the president and vice president, familiar to the public by their sunglasses and earphones and through films such as "In The Line Of Fire" starring Clint Eastwood in 1993 and "Guarding Tess" a year later with Shirley MacLaine and Nicolas Cage.
The Secret Service was created as a division of the U.S. Treasury on July 5, 1865, to suppress counterfeit currency.
It quickly expanded to take on the Ku Klux Klan, moonshiners and others "perpetrating fraud against the government," according to the Secret Service website www.secretservice.gov.
The agency assumed responsibility for protecting the president in 1902, and its role continued to expand in ensuing decades. When the Department of Homeland Security was created after the September 11, 2001, terrorist attacks, the Secret Service moved over from Treasury.
Sullivan has an exemplary record with the Secret Service, rising up through the ranks and receiving the Distinguished Presidential Rank Award in 2005 and 2010, the website said.
Born in Arlington, Massachusetts, Sullivan graduated from Saint Anselm College in New Hampshire and joined the Secret Service as a special agent in Detroit in 1983. Among his assignments were stints in the fraud division and the presidential protective division before becoming a supervisor.
The Colombia scandal is not the first public problem encountered under Sullivan's direction. In 2009, a couple got into a White House state dinner without invitations, embarrassing the Secret Service. While the agency had ultimate responsibility for security that night, the problem was eventually attributed more to a White House failure to check invitations.
Sullivan's challenge now is to make sure that the agency's reputation of excellence as an elite force continues, Cummings said.
"It's just not what happened here, but it's even the appearance that that organization could be pierced," he said. "We don't even want people to even be thinking that that's possible. And so they've got to re-establish trust, and they also have to make sure that any holes they may find, any weakness, will be filled."
CNN's Dana Bash contributed to this report.