"He's done more to bring our education system -- sometimes kicking and screaming -- into the 21st century more than anybody else," Obama said at a White House ceremony. "America is going to be better off for what he's done."
Duncan leaves the administration as the longest-serving education secretary in U.S. history, Obama said.
"It's a record that I truly believe that no other education secretary can match," he said. "Arne bleeds this stuff. He cares so much about our kids. And he's been so passionate about this work."
While speaking about the impact his parents -- both educators -- had on him, Duncan got emotional.
"All our lives we saw what kids could do when given a chance, and that's why we do this today," he said.
Obama has selected Deputy Secretary of Education John B. King, Jr. to replace him.
In an internal memo to Department of Education staff, Duncan cited the commute between his home and family in Chicago in announcing his departure and called his job the "greatest honor of my life."
"We have been lucky to have an a amazing team here from Day One, but I honestly believe our team today is the strongest it's ever been," Duncan wrote. "So it's with real sadness that (I) have come to recognize that being apart from my family has become too much of a strain, and it is time for me to step aside and give a new leader a chance."
Obama brought Duncan with him to Washington from Chicago, where he served as the city's schools chief. He's long been one of the president's closest friends in the administration, their relationship forged in the days before either came to national prominence.
Duncan, along with Obama's first chief of staff, Rahm Emanuel, and senior advisers Valerie Jarrett and David Axelrod, formed a hometown crowd for the Obamas when they arrived to the White House. Jarrett remains in her post at the White House, though Axelrod and Emanuel have both returned to Chicago -- Emanuel as the city's mayor, and Axelrod as chair of a politics institute at the University of Chicago.
Duncan said he had not made a decision about his next steps besides spending time with his family and added that he expected to find work that involved "expanding opportunity for children."
Duncan also offered effusive praise for King.
"Over the years that I have known him, and especially in the months we have worked together here, I've come to recognize John as one of the most passionate, courageous, clear-headed leaders in our field," Duncan said.
Duncan's departure means Agriculture secretary Tom Vilsack will be the sole remaining Cabinet-level secretary who has been with Obama since 2009. Obama's Office of Management and Budget Director Shaun Donovan started in 2009 as his Housing and Urban Development secretary.
In the early days of the administration, Duncan and Obama often played pickup basketball on the weekends in Washington — Duncan is among the group of close Obama confidantes who participated in the President's Election Day pickup game in 2012 -- though Obama's days on the court have waned in favor of days on the golf course.
As education secretary, Duncan pushed reforms to improve the quality of education, including advocating for the controversial Common Core requirements. Many Republicans say the requirements provide unnecessary federal interference in school curriculum.
He also implemented the administration's "Race to the Top" funding program that has states compete for federal school funding, as well as gone after for-profit colleges for predatory practices.