Carson, who was confirmed to lead the department earlier this month, heralded the work ethic of immigrants before implying slaves who came to the United States worked harder than others.
"There were other immigrants who came in the bottom of slave ships, who worked even longer, even harder, for less, but they too had a dream that one day their sons, daughters, grandsons, granddaughters, great grandsons, great granddaughters might pursue prosperity and happiness in this land," said Carson, who is black.
"And do you know of all the nations in the world, this one, the United States of America, is the only one big enough and great enough to allow all those people to realize their dream. And this is our opportunity to enhance that dream," he added.
Earlier in the remarks, Carson said: "That's what America is about, a land of dreams and opportunity."
HUD spokesman Brian Sullivan attempted to clarify Carson's statement, saying, "Nobody here believes he was equating voluntary immigration with involuntary servitude."
Critics quickly decried his comment.
"Ben Carson is also the guy who once compared Obamacare to slavery," tweeted Keith Boykin, a CNN political contributor. "I'm starting to think he may not understand the word 'slavery.'"
The NAACP declined to comment, but the group tweeted: "Immigrants?" in response to a story about Carson's comments.
Amid an uproar over his remarks, a statement was posted on Carson's official Facebook page late Monday that said slaves and immigrants had "two entirely different experiences."
"Slaves were ripped from their families and their homes and forced against their will after being sold into slavery by slave traders. The Immigrants made the choice to come to America."
"The two experiences should never be intertwined, nor forgotten," according to his Facebook post.
The worst thing 'since slavery'
This is not the first time Carson has likened something to slavery.
In 2013, Carson said that Obamacare -- the Obama administration's landmark healthcare law -- was the worst thing "since slavery."
"You know Obamacare is really, I think, the worst thing that has happened in this nation since slavery," Carson said at the Values Voter Summit in Washington. "And it is in a way, it is slavery in a way, because it is making all of us subservient to the government, and it was never about health care. It was about control."
Carson also compared abortion to slavery in an interview with NBC during his 2016 presidential run.
"During slavery -- and I know that's one of those words you're not supposed to say, but I'm saying it -- during slavery, a lot of the slave owners thought that they had the right to do whatever they wanted to the slave," Carson said in October 2015. "What if the abolitionists had said, 'I don't believe in slavery, I think it's wrong, but you guys do whatever you want to do?"