Carson had suggested earlier on Tuesday that if elected president he might be the first African-American person to hold the position. And speaking to CNN's Poppy Harlow on "CNN Newsroom" later that afternoon, Carson suggested Obama's upbringing did not reflect the experiences of many other African-Americans.
"I grew up in Detroit, and I grew up in Boston. In Boston, we lived in the ghetto. There were a lot of violent episodes there. There were rats, there were roaches. It was dire poverty," Carson said, speaking of his mother who worked around-the-clock but refused to accept welfare. "Now, let me contrast that to the President, who went to private schools, grew up in a relatively affluent environment, had an opportunity to live in multiple cultures and different countries. I think that's a very different experience."
Carson maintained that he was not "criticizing" Obama. But he had gone further in an interview with Politico's Glenn Thrush on his "Off Message" podcast that aired Tuesday
"He's an 'African' American. He was, you know, raised white," the retired neurosurgeon said. "So, for him to, you know, claim that, you know, he identifies with the experience of black Americans, I think, is a bit of a stretch."
Obama was born in Honolulu to a white American mother and a Kenyan father. Carson was born in Detroit to two black American parents.
"Like most Americans, I was proud that we broke the color barrier when he was elected, but I also recognize that his experience and my experience are night-and-day different. He didn't grow up like I grew up by any stretch of the imagination," Carson told Thrush. "Not even close."
California Rep. Darrell Issa, who has endorsed Sen. Marco Rubio for president, said Carson's "loose talk" proves that he is not ready for the White House.
"This is the reason that political loose talk by people who are just beginning to do it is always a problem in real statesmanship," he said Tuesday on CNN's "New Day." "The fact is that Dr. Carson is not really ready to represent America around the world."
Carson also said that "very little" criticism of Obama has to do with the President's race.
"You have to recognize that what President Obama represents is an ideology that is antithetical to the ideology of most people in the Republican Party. And I don't think it has anything to do with race," he said. "I mean, Hillary (Clinton) represents that kind of ideology also, and they'll say it's because she's a woman. I mean, any guy who represents that kind of ideology is going to evoke exactly the same types of criticism."
Most of the racism Carson experiences comes from progressives, he said.
"I think the way that I'm treated, you know, by the left is racism," Carson said. "Because they assume because you're black, you have to think a certain way. And if you don't think that way, you're 'Uncle Tom,' you're worthy of every horrible epithet they can come up with; whereas, if I weren't black, then I would just be a Republican."
Carson, who placed sixth in South Carolina's Republican primary last Saturday, said he's never experienced racism from the Republican Party.
"I don't find any particular problem being an African-American in the Republican Party. The people -- I know that in the progressive side of things, they like to say that the Republicans are racist. I know that. I haven't experienced that," he said.