Rep. John Lewis, who helped lead the civil rights march in Selma, suggested Scalia should consider recusing himself from the case, Fisher v. University of Texas.
"Justice Scalia's evident bias is very troubling to me. It leads me to question his ability to make impartial judgments in this case," the Georgia Democrat said in a statement.
Lewis was one of the top leaders of the civil rights movement in the '60s and has served in Congress nearly 30 years.
Scalia was referring to a brief filed in the case that discussed an academic proposition called "mismatch theory."
"I'm just not impressed by the fact that the University of Texas may have fewer (African-Americans)," Scalia said, in part. "Maybe it ought to have fewer. And maybe some -- you know, when you take more, the number of blacks, really competent blacks, admitted to lesser schools, turns out to be less."
"There are those who contend that it does not benefit African-Americans to get them into the University of Texas where they do not do well, as opposed to having them go to a less-advanced school, a less -- a slower-track school where they do well," Scalia said, according to the court transcript. "One of the briefs pointed out that most of the black scientists in this country don't come from schools like the University of Texas."
The brief Scalia referred to said, "Students with an interest in science who are admitted to a very competitive school via a large preference tend to drop out of the sciences at a much higher rate than do otherwise similar students who attend somewhat less competitive programs."
"Competition mismatch appears to be a major factor in the low rate at which African-American students become scientists, despite high levels of interest in the sciences," it said.
Lewis said he was "shocked and amazed" by Scalia's remarks.
"His suggestion that African-Americans would fare better at schools that are 'less advanced' or on a 'slow-track' remind me of the kind of prejudice that led to separate and unequal school systems -- a policy the Supreme Court declared unconstitutional decades ago," Lewis said.
Senate Minority Leader Harry Reid took to the Senate floor to decry Scalia's comments as "racist" -- and sought to tie him to Donald Trump and Republicans.
Reid read Scalia's full comments on the Senate floor Thursday morning and tore them apart.
"These ideas that he pronounced yesterday are racist in application, if not intent," Reid said. "I don't know about his intent, but it is deeply disturbing to hear a Supreme Court justice endorse racist ideas from the bench on the nation's highest court. His endorsement of racist theories has frightening ramifications, not the least of which is to undermine the academic achievements of Americans, African-Americans especially."
Reid also tied the comments to rhetoric coming from Trump, the GOP presidential front-runner, and the broader Republican field.
"As we speak, Donald Trump is proposing to ban Muslim immigration. Other leading candidates are proposing a religious test, tossing around slurs on a daily basis," Reid said. "And now a Republican-appointed justice endorsing racist ideas from the Supreme Court bench. The only difference between the ideas endorsed by Trump and Scalia is that Scalia has a robe and a lifetime appointment. Ideas like this don't belong on the Internet, let alone the mouths of national figures."
Reid called Scalia "out of touch" with the nation's ideals and called his comments "distressing," saying they were a reminder of a need for vigilance to protect opportunity for Americans.
"The idea that African-American students are somehow inherently intellectually inferior to other students is despicable," Reid said. "It's a throwback of time to a time that America left behind half a century ago. The idea we should be pushing well-qualified African-Americans out of the top universities into lesser schools is unacceptable."
Reid's diatribe lasted for about five minutes of the Senate's period of morning business. Both Reid and his counterpart, Senate Majority Leader Mitch McConnell, often use the daily time to speak on issues of concern and make political points.
The White House had a fairly muted response, saying only President Barack Obama has a different view than Scalia.
"The sentiment he expressed is not something that anyone who has heard the public comments of senator and now President Obama -- they would detect a difference," press secretary Josh Earnest said in Thursday's briefing. "I think the comments articulated by Justice Scalia represent quite a different view than the priorities and values President Obama has spent his career talking about."