(CNN) -- Critics grilled him on issues such as presidential authority and negotiating with terrorists, but Defense Secretary Chuck Hagel kept returning to the human side of the debate over a deal that freed Bowe Bergdahl.
"We got our one remaining prisoner back," Hagel said near the end of more than five hours of testimony at the first congressional hearing on the exchange of Bergdahl, an Army sergeant held captive in Afghanistan, for five Taliban figures detained at Guantanamo Bay. "I don't think that's an incidental accomplishment."
The May 31 swap ignited controversy in the partisan frenzy permeating Washington in an election year, and Wednesday's House Armed Services Committee hearing devolved at times into political marksmanship seemingly aimed more at creating sound bites and fodder for new ads than promoting further understanding of the issue.
Hagel appeared frustrated and even angry at times, but stayed on-message after an opening statement that laid out the legal and ideological foundation for the agreement that the Taliban claim as a victory.
"War, every part of war, like prisoner exchanges, is not some abstraction or theoretical exercise," Hagel said early on, presenting a them he repeated throughout the hearing. "The hard choices and options don't fit neatly into clearly defined instructions in 'how to' manuals. All of these decisions are part of the brutal, imperfect realities we all deal with in war."
'We complied with the law'
Speaking of the Bergdahl deal approved by President Barack Obama, Hagel declared "we complied with the law, and we did what we believed was in the best interests of our country, our military, and Sergeant Bergdahl."
"The President has constitutional responsibilities and constitutional authorities to protect American citizens and members of our armed forces. That's what he did," Hagel said. "America does not leave its soldiers behind. We made the right decision, and we did it for the right reasons -- to bring home one of our own people."
Republican critics contended the cost was too high because it freed hardened terror suspects to possibly rejoin the fight against U.S. interests, especially in return for a soldier accused by some in his unit of deserting his post.
Legislators from both parties also complained that the Obama administration failed to notify Congress ahead of time, as required by the National Defense Authorization Act for transfers of Guantanamo detainees.
Republican Rep. Buck McKeon of California, the committee chairman, labeled as "misleading and at times blatantly false" the explanations by White House officials at a classified briefing on Monday for making the exchange and not notifying Congress beforehand.
'We could have done a better job'
Hagel conceded the administration should have tried harder to let Congress know about what he called rapidly moving developments, saying he understood the frustration caused by the speed of the mission.
"We could have done a better job of keeping you informed," Hagel said, noting that only a few hours elapsed between making the final arrangements with the Taliban through the government of Qatar, which mediated the agreement.
"As the opportunity to obtain Sergeant Bergdahl's release became clearer, we grew increasingly concerned that any delay, or any leaks, could derail the deal and further endanger" him, Hagel said.
"We were told by the Qataris that a leak, any leak, would end the negotiation for Bergdahl's release. We also knew that he would be extremely vulnerable during any movement, and our military personnel conducting the handoff would be exposed to a possible ambush or other deadly scenarios in very dangerous territory," he said.
Under the swap, the released Taliban detainees were taken to Qatar and will remain there for a year, with the Qatari government saying it will keep an eye on them.
McKeon worried that the newly freed Taliban figures "still pose a threat to Americans and Afghans alike" because "in a year, they will be free to return to Afghanistan."
Other GOP legislators accused the administration of acting on its own to evade opposition to the deal in Congress, as well as trying to hide a shift in U.S. policy by negotiating with terrorists to secure Bergdahl's release. They seized on Hagel's confirmation that Bergdahl at times was held by the Haqqani network, a Taliban wing designated a terrorist group in 2012.
Negotiating with terrorists?
Hagel repeatedly said the on-off negotiations dating back to 2011 involved the broader Taliban that formerly governed Afghanistan, and that the government of Qatar served an intermediary in the final stage of talks in April and May.
"We didn't negotiate with Haqqani," Hagel said, adding that Bergdahl was moved around during his five years of captivity. That brought accusations of deceptive reasoning, with Rep. Mike Turner of Ohio challenging Hagel in a particularly sharp confrontation.
"How is it the United States could've been in negotiations with the Haqqani network, a listed terrorist organization, and it not conflict with our policy that we do not negotiate with terrorists?" Turner asked, to which Hagel replied that "we dealt directly with the government of Qatar."
Turner shot back: "So now, the new policy of this administration is, 'we don't negotiate with terrorists directly.'"
The next legislator up, Democratic Rep. Jim Cooper of Tennessee, criticized Turner for being "prosecutorial" and wondered aloud if his Republican colleague might be running for House GOP leader after the primary defeat Tuesday of Eric Cantor of Virginia.
"I'm shocked, really, that this has become such a political football," Cooper said.
Hagel also complained about the politicization of the Bergdahl exchange and said he was "offended and disappointed in how the Bergdahl family has been treated by some people" amid the controversy over the release.
The FBI is investigating threats against the family following the appearance by Bergdahl's parents with Obama at the White House for the announcement of their son's release.
In another heated moment that left Hagel visibly angry, Republican Rep. Jeff Miller of Florida badgered him about why it was taking the military so long to interview Bergdahl, who is being treated under a post-captivity regimen at the Landstuhl Medical Center in Germany.
When Hagel started explaining that doctors treating Bergdahl will decide when he is ready to talk, Miller cut him off.
"Wait a minute. Wait a minute. Why hasn't he been returned to the United States?" Miller asked, noting that wounded soldiers from battle zones get flown back for treatment within days. "You're trying to tell me that he's being held at Landstuhl, Germany, because of his medical condition?"
Hagel responded: "Congressman, I hope you're not implying anything other than that," and the two continued to talk over each other.
"I don't like the implication of the question," Hagel said at one point, and when Miller persisted about injured combat troops coming back to the United States quickly, Hagel raised his voice.
"This isn't just about a physical situation," he said. "This guy was held for almost five years in God knows what kind of conditions. ... This is not just about can he get on his feet and walk and get to a plane."
Shortly afterward, Democratic Rep. Jackie Speier of California looked over at the Republican side of the committee room and asked for everyone "to think for a moment how we would be responding if Bowe Bergdahl was our son."
"I really fear for his return to this country with the kind of rhetoric that is being spewed in this very room," she said.
Members of Bergdahl's unit have claimed that U.S. soldiers died on missions trying to find him after he disappeared in 2009, but Hagel said Wednesday that he had seen "no evidence that directly links any American combat death to the rescue or finding or search of Sergeant Bergdahl."
Hagel's testimony provided some added detail but otherwise hewed to previous statements by administration officials from Obama on down about what led to the exchange and how it was carried out.
He noted that original talks dating to 2011 involving the possible release of Taliban detainees at Guantanamo involved six people, but one died, and the United States previously rejected Taliban demands for letting go the remaining five.
Timeline of talks
In a timeline provided by Hagel, he said contacts with the Taliban led the United States to request a proof-of-life video of Bergdahl last November. The video, received in January, was disturbing and "showed a deterioration in his physical appearance and mental state," Hagel said.
"Our entire intelligence community carefully analyzed every part of it and concluded that Sergeant Bergdahl's health was poor and possibly declining. This gave us growing urgency to act," he said.
The United States signed a memorandum of understanding with Qatar on May 12 regarding how Qatar would handle Taliban detainees transferred to its custody in the possible exchange.
Soon after, "U.S. officials received a warning from the Qatari intermediaries that as we proceeded, time was not on our side. ... This indicated that the risks to Sergeant Bergdahl's safety were growing," Hagel said.
Final arrangements were worked out on May 27 and the swap took place four days later, with the last-minute uncertainty causing the administration to decide against notifying Congress, he added.
The administration held two classified briefings this week for members of Congress -- one to the Senate Armed Services Committee on Tuesday and Monday's session with House members.
Those meetings intensified the rhetoric on Capitol Hill.
Rep. Adam Schiff, a California Democrat, told CNN's "Erin Burnett OutFront" that White House officials should have told congressional leaders, even if the law gave them wiggle room.
"I think the President had the constitutional authority ... to make this decision without consulting with Congress," Schiff said. "But I think it would have been wiser, far wiser, for the administration to have notified, certainly the leadership of Congress in the interest of having good relations."
House Speaker John Boehner said that he was briefed ahead of another significant and secret military mission -- the 2011 raid that killed al Qaeda leader Osama bin Laden.
"I was given a heads-up several days before" the bin Laden mission, Boehner said. "So this idea that they couldn't trust us to not leak things is just not true."
He also expressed concern of the security risk, telling reporters Tuesday he had no doubt "that there are going to be costs, lost lives associated with what came out of this."
A war captive, not a hostage
National Security Council spokeswoman Caitlin Hayden said the administration had received "assurances from Qatar" during the deal about monitoring the five released Taliban figures.
"We have acknowledged that some of these individuals could attempt to return to activities that are of concern. The President said so explicitly on his trip last week," she said. "But we have confidence that we will be in a position to go after them if, in fact, they are engaging in activities that threaten our defenses."
Missing since 2009
Bergdahl went missing on June 30, 2009, in Afghanistan's Paktika province, where he was deployed with the 1st Battalion, 501st Infantry Regiment, 4th Brigade Combat Team, 25th Infantry Division.
An Army fact-finding investigation conducted in the months after his disappearance concluded he left his outpost deliberately and of his own free will, according to an official who was briefed on the report.
The Army has no definitive finding that Bergdahl deserted because that would require knowing his intent -- something Army officials couldn't learn without talking to the soldier, a U.S. military official told CNN.
Bergdahl is "continuing to improve every day" as he recovers at a military hospital in Germany, Pentagon spokesman Col. Steve Warren said this week, but he hasn't spoken with his family yet.
When Bergdahl is ready, he will be flown to the San Antonio Military Medical Center in Texas, where he may be able to reunite with his parents.
CNN's Michael Pearson, Mariano Castillo, Paul Steinhauser, Catherine Shoichet, Virginia Nicolaidis, Kevin Liptak, Barbara Starr, Elise Labott, Qadir Sediqi, Dana Bash and Deirdre Walsh contributed to this report.