Nihad Awad, executive director of the Council on American-Islamic Relations, said the group is calling on Carson to "withdraw from the presidential race because he is unfit to lead, because his views are inconsistent with the United States Constitution."
Carson is currently a top contender for the GOP presidential nomination, taking third place with 14% of support in the latest CNN/ORC poll, released Sunday.
His comments came as he responded to another controversy involving a GOP presidential candidate and Muslims in America when a questioner told Donald Trump at a campaign event in New Hampshire that "we have a problem in this country ... called Muslims" and said that President Barack Obama is a Muslim.
Awad insisted that it was Carson's, not Muslim Americans', views that are inconsistent with the Constitution, as the document states that there shall be "no religious test" for candidates seeking public office.
Awad said he was "shocked" to hear Carson's comments, which he described as "anti-Muslim." He urged the public and other presidential candidates to "repudiate" Carson's views.
"The protection of freedom of religion in America is a fundamental principle of our country so whether you are Christian, you are Jewish, you are Muslim, you are Catholic, you are black, you are brown, you are white -- if you are born in this country, if you uphold Constitution, if you have the vision, if you are fit to lead, you can lead irrespective of your faith tradition," Awad said.
The Anti-Defamation League, a group that fights anti-Semitism and other bigotry, called Carson's comments "deeply offensive, un-American and contrary to the Constitution" in a statement from ADL National Director Jonathan Greenblatt.
"As the campaign season advances, we urge all presidential candidates to avoid innuendo and stereotyping of all sorts, including against people based on their faith, particularly American Muslims and, instead, to confront all forms of prejudice and bigotry. Remarks suggesting that all Muslims follow extremist interpretations of Islam have no basis in fact and fuel bigotry," Greenblatt said.
Greenblatt also knocked Trump for his "failure to stand up to an anti-Muslim bigot at a campiagn rally who questioned whether President Obama was a Muslim."
Immediately following Awad's comments, Carson's business manager, Armstrong Williams, told CNN that Carson was not trying to "hurt or offend" but came to his conclusion about Islam "because of his love for America."
"What an individual believes impacts how America is governed and what we become as a nation," Williams said, adding that Carson is not alone in his view of Islam.
In fact, according to a Gallup survey published in June
, 38% of Americans polled said they would not support a Muslim candidate for president, while 60% said they would.
"There are many Americans, if they search their hearts, they believe the same thing -- they don't want to believe it but when they look at the world and look at how Islam has become polluted and what is happening in Europe and in the Middle East and around the world, you have to take time to pause and ask what is happening around the world?" Williams said.
Williams also slammed what he called the "destructive tenets of Islam and Sharia law."
Democratic front-runner Hillary Clinton tweeted a brief question and answer Monday: "Can a Muslim be President of the United States of America? In a word: Yes. Now let's move on," signing the tweet "H" to show it was from her.
Republican front-runner Donald Trump declined to condemn Carson's comments as he sought to extinguish the controversy surrounding comments about Muslims swirling around his own candidacy.
Trump said in his case he assumed the questioner who said at his campaign event that Muslims were a problem in America was talking about "radical Islam," but has refused to disavow the questioner's comments about Obama.
Of Carson, Trump said the retired neurosurgeon is "speaking his opinion" and "feels very strongly about it."
Most other GOP presidential candidates who weighed in have offered veiled or indirect criticism of Carson's remarks.
Sen. Ted Cruz of Texas pointed to the Constitution.
"You know, the Constitution specifies there shall be no religious test for public office and I am a constitutionalist," Cruz said
Lousiana Gov. Bobby Jindal slammed the press for playing a "dumb game," but said he would support a Muslim candidate running as a Republican who would respect "the Judeo-Christian heritage of America," will commit to fight ISIS and radical Islam and will "condemn cultures that treat women as second class citizens."
But Jindal added one more caveat: The candidate must swear the oath of office "on the Bible." A Jindal spokesman declined to say why a Muslim candidate would need to swear the oath of office on a Bible, not a Koran.
Sen. Lindsey Graham of South Carolina took the most direct shot at Carson, saying he is "not ready to be commander in chief."
"America is an idea, not owned by a particular religion," Graham tweeted Sunday. "@RealBenCarson needs to apologize to American Muslims. He is a good doctor, but clearly not prepared to lead a great nation."