Philadelphia (CNN) -- Attorney General Eric Holder is most afraid of a terrorist attack on the United States from a so-called "lone wolf," such as the attackers involved in the Washington Navy Yard and Los Angeles International Airport shootings.
"I'm very concerned about individuals who get radicalized in a variety of ways, sometimes self-radicalized," the nation's top law enforcement officer told CNN's Justice reporter Evan Perez in an exclusive interview.
Just days after a shooter entered LAX airport and killed one Transportation Security Administration worker, Holder rejected the idea to arm those officers.
"Now, that doesn't mean that we shouldn't review the measures that are in place to keep people safe from the time that they get out of their cars and go into the terminals," he said.
In the wide-ranging conversation Tuesday, he also said that he rejected demands from supporters that NSA leaker Edward Snowden be offered clemency.
As President Barack Obama's top legal expert, the attorney general deflected a question about whether he has gotten a free pass for approving many of the surveillance tactics at the National Security Agency.
He responded: "We have to ask some very legitimate questions."
Holder spoke to CNN during a visit to a federal courthouse Tuesday to highlight a program that helps ex-offenders reintegrate into their communities.
Holder said he didn't support clemency for Snowden, the national security contractor who leaked the classified information about surveillance programs.
He said "the mechanisms that he used" to publicize his concerns with the government's surveillance aren't "worthy of clemency."
"I think that he has clearly broken the law and harmed the nation that he claims to have loved," Holder said.
While Holder said Snowden deserved to be punished for his crime, in nearly the same breath he said that Snowden sparked a necessary conversation.
"The conversation that we are engaged in is one that I think is certainly worthwhile, to try to determine how do we safeguard privacy and keep the American people safe," Holder said.
Holder says the administration needs to ask itself if the surveillance net had "gone to far," but it wouldn't go as far as saying if it was a concern he had before the Snowden incident.
"I didn't say that," Holder responded, adding that the administration was already having "conversations" about the collection of metadata.
"It was a conversation that, frankly, was going on, certainly within the administration," Holder said. "Now the conversation is a .. is a more public one."
The reach of the NSA's surveillance has caused a fiery response overseas and domestically since Snowden leaked documents to The Washington Post and The Guardian newspapers.
Holder said Snowden could have addressed his concerns with government surveillance in other ways, such as through the court system.
Throughout his tenure, Holder has come under fire from Republican critics who held him in contempt of Congress. He has been under investigation, and subsequently cleared by the inspector general, in the botched gun-trafficking program "Fast and Furious." He has also taken fire for his role in defending the Voting Rights Act and reviewing cases of detainees at Guantanamo Bay.
Despite the turmoil, Holder has refused calls for his resignation.
As instances of mass gun violence continue to spark conversations about gun control, Holder said Tuesday that politicians in Washington failed to pass meaningful gun safety legislation.
"Yes, we've failed. We've failed," Holder said, referring to Congress' refusal to pass stricter background checks of gun purchasers after the mass shooting at Sandy Hook Elementary School in Newtown, Connecticut, last year.
"It can't be the new normal. It cannot be something that we accept. Uh, there is too much gun violence," he added.
As two states have legalized marijuana and voters in one of those state, Colorado, head to the polls today to vote on a marijuana tax, Holder said he doesn't anticipate any change to federal laws outlawing the drug.
Instead, he said he is looking at how marijuana laws are enforced.
While Holder plans to stay on the job, he admits that times haven't always been easy.
"I've had some days that have been better than others," he said.