Several justices expressed deep concern about the testimony, suggesting that they are poised to rule in favor of the inmate, Duane Buck.
"What occurred at the penalty phase of this trial is indefensible," said Justice Samuel Alito who called the testimony "bizarre."
"What competent counsel would put that evidence before a jury?" asked Justice Ruth Bader Ginsburg.
Justice Stephen Breyer queried whether there was "some good reason" Buck shouldn't be able to re-open his case.
Buck has been on death row for the 1995 murders of Debra Gardner and Kenneth Butler. He is not arguing his innocence, but is asking for a new sentencing hearing because his own trial counsel was ineffective.
Buck's case has been caught in procedural knots for years, and Texas argues in part that his lawyers failed to raise his claim in a timely manner.
The case is one of the first high-profile cases of the Supreme Court's new term and comes as the country and the court grapple with the issue of racial bias in the criminal justice system.
The legal issue before the court is whether Buck's case meets the "extraordinary circumstances" test justifying the reopening of his sentencing.
At the heart of the matter is testimony provided by Dr. Walter Quijano, one of two psychologists retained by the defense. Quijano tesfitied that the fact that Buck was black "increased the probability" that he would commit future acts of criminal violence. In Texas, so-called "future dangerousness" must be established before a death sentence is rendered.
"Put another way," Buck's current lawyers argue in court papers, "Mr. Buck's lawyers presented evidence that Mr. Buck was more deserving of a death sentence under Texas law because of his race." They say such prejudicial evidence is the "epitome of ineffective assistance of counsel."
They point out that the state itself eventually conceded error in six other cases where Quijano's testimony had been elicited.
Wednesday, Texas Solicitor General Scott Keller drew a distinction between Buck's case and the others because it was the defense itself who called Quijano and elicited race-related testimony on direct examination.
But the justices seized on that distinction. Justice Sonia Sotomayor asked why it should matter which side elicited the issue of race. She was joined by Chief Justice John Roberts and Justice Elena Kagan on that question.
The justices did, however, struggle with where to draw a line regarding when an inmate meets the criteria necessary to have his or her case opened up. Alito worried about other inmates "opening the door" and asking to raise ineffective counsel claims. Roberts worried that Buck's "unique case" was not the proper platform to issue general rules.
Keller began his time at the podium addressing the circumstances surrounding Buck's crime, including the fact that he murdered his ex-girlfriend, Gardner, in front of her children.
Keller pointed to the fact that he also shot his step-sister at point blank range and shot another man through the heart.
But he was cut off, as the justices circled back to the race-based testimony.
Keller also argued that Quijano's testimony played a limited role at trial and other evidence of his future dangerousness came from the brutality of the murders, Buck's lack of remorse after he was apprehended as well as the testimony of another ex-girlfriend.
In most cases, supporters of the parties file so called "friend of the court briefs" in support of the party they believe has the stronger legal argument.
There are no such briefs filed in support of Texas perhaps in part because the case is so fact specific to Buck's circumstances.
A group called the National Black Law Students Association, however, filed a brief in support of Buck emphasizing how the group believes the case impacts race relations today.
"When an expert witness told the jury that Mr. Buck was dangerous because he is Black, he dredged up into the open for all members of the jury to see the monstrous specter that is never far from the surface: the violent Black brute, the single most fearful, dehumanizing, and cruel stereotype Black people have had to endure," wrote the group's lawyer, Deborah N. Archer.