(CNN) -- The inside accounts of the Obama administration's internal debate over Afghanistan may be grabbing the headlines, but former Defense Secretary Robert Gates' new memoir has some other eye-openers as well.
Yes, Gates levels blistering criticism at Congress, Vice President Joe Biden and former colleagues and describes White House aides working to undercut President Barack Obama's confidence in his own commanders. But in "Duty: Memoirs of a Secretary at War," the low-key Gates -- who replaced Donald Rumsfeld at the Pentagon late in George W. Bush's administration and was kept on by Obama when he took office in 2009 -- also talks about his own moments of doubt, anger and frustration. Here are a few items from the book that may surprise readers.
-- Take this job and shove it: Gates fantasized about quitting
"All too frequently, sitting at that witness table, the exit lines were on the tip of my tongue: 'I may be the secretary of defense, but I am also an American citizen, and there is no son of a bitch in the world who can talk to me like that. I quit. Find somebody else.' It was, I am confident, a widely shared fantasy throughout the executive branch."
-- Bush and Obama 'had much more in common than I expected.' That wasn't always good
"Both were most comfortable around a coterie of close aides and friends (like most presidents) and largely shunned the Washington social scene. Both, I believe, detested Congress and resented having to deal with it, including members of their own party. And so, unfortunately, neither devoted much effort to wooing or even reaching out to individual members or trying to establish a network of allies, supporters -- or friends," Gates writes. "They both had the worst of both worlds on the Hill: they were neither particularly liked nor feared. Accordingly, neither had many allies in Congress who were willing to go beyond party loyalty, self-interest, or policy agreement in supporting them."
However, Gates adds, "I liked and respected both men."
"I witnessed both of those presidents make decisions they believed to be in the best interest of the country regardless of the domestic political consequences, both thereby earning my highest possible respect and praise. Although, as I've said, political considerations were far more a part of national security debates under Obama, time and again I saw him make a decision that was opposed by his political advisers or that would be unpopular with his fellow Democrats and supportive interest groups."
-- The price of public service:
Gates writes that he spent $40,000 on lawyers to help him fill out financial disclosures and other government paperwork. Before his 2006 confirmation hearing, he filled out a 65-page questionnaire from the Senate Armed Services Committee. At one point, Bush's chief of staff, Josh Bolten, "asked if I had any ethical issues that could be a problem, like hiring illegal immigrants as nannies or housekeepers.
"I decided to have some fun at his expense and told him we had a non-citizen housekeeper. Before he began to hyperventilate, I told him she had a green card and was well along the path to citizenship. I don't think he appreciated my sense of humor," Gates wrote.
He told his wife, "I have to do this, but I just hope I can get out of this administration with my reputation intact."
-- 'I couldn't sleep that night'
Gates writes that an encounter with a woman in a restaurant who had two sons fighting in Iraq drove home the human dimension of the job. "I couldn't finish my dinner, and I couldn't sleep that night," he wrote. "Our wars had just become very real to me, along with the responsibility I was taking on for all those in the fight."
-- Time to go:
By early 2011, "My fuse was really getting short. It seemed like I was blowing up -- in my own, quiet way -- nearly every day, and no longer just in the privacy of my office with my staff ... I had blown up at (National Security Adviser Thomas) Donilon and the vice president at a meeting on Libya on March 2 and at House Defense Appropriations chair Bill Young on the third, and had come close to openly arguing with the president in the NSC meeting that same day, and had gone off on Donilon again on the fifth. Partly, I think, I was just exhausted from the daily fights."
In addition, he wrote that his desire to protect U.S. troops -- "from the wars we were in and from new wars -- was clouding my judgment and diminishing my usefulness to the president, and this is played a part in my decision to retire." He left that June, with Obama awarding him the Presidential Medal of Freedom -- the highest U.S. civilian honor -- on his way out.