Washington (CNN) -- Virginia GOP Rep. Randy Forbes on Wednesday became the first member of Congress to call for U.S. Secret Service Director Mark Sullivan to step down due to a prostitution scandal involving eleven agents that has raised questions about security at the agency.
"There are only so many strikes you get -- in baseball it's three," Forbes told CNN. "I think (Sullivan's) had three. I think it's time to put somebody else in there to make sure we're getting a different culture in the Secret Service."
Forbes, a senior member of the House Armed Services Committee, cited an earlier security breach at a White House state dinner involving the Salahis, a couple who got through security without an invitation. He also noted the agency's recent inability to remain within its budget.
There needs to be "change at the top," Forbes stressed.
But House Armed Services Committee Chairman Buck McKeon, R-California, disagreed with Forbes, telling reporters he didn't think the head of the agency should be held responsible.
"I think that's a reach ... to go after the head of an organization because a few guys mess up," McKeon said Wednesday.
Forbes said the security issues were serious, and Congress needed to be more involved in finding out what happened in Colombia.
"We aren't just talking about one prostitute. We are talking about a bus load of prostitutes that they're bringing in," Forbes said.
Eleven Secret Service agents are allegedly involved in the scandal, along with several members of the U.S. military. The agents were in Colombia in advance of a visit by President Barack Obama.
The alleged prostitutes, the youngest of whom were in their early 20s, had all signed in at Cartagena's Hotel El Caribe, flashing their local IDs. One of the women, a source said, became involved in a dispute with Secret Service personnel about how much she was to be paid. That dispute brought the incident to light.
"This is kind of security planning 101," Forbes said. "It is the basic core fundamental, and the big concern I have is that it isn't just Americans looking into this. It is the rest of the world, and so I think it is time we jump on this and do something about it."
The Virginia Republican said he is confident the Pentagon is capable of handling its component of the investigation -- as many as ten members of the military have been implicated in the scandal. But Forbes doesn't have the same level of trust in agencies outside the military, and said he planned to urge the head of the House Judiciary Committee to hold hearings into the matter.
The Senate Judiciary Committee, meanwhile, has announced it will hold a hearing on the matter next Wednesday. Homeland Security Secretary Janet Napolitano has been invited to testify. The Secret Service is part of the Homeland Security Department.
Like Forbes, McKeon believes that the Pentagon is "very capable" of doing its own probe of members of the military who may be involved. He doesn't see a role for Congress in the matter.
"I don't think Congress needs to jump in to start trying to fix something that we don't even know what the problem is yet," McKeon said.
But McKeon said he would likely raise the issue with Defense Secretary Leon Panetta when he meets with Panetta before the secretary's scheduled appearance before the House Armed Services Committee on Thursday.