(CNN) -- A Florida pastor who canceled plans for his congregation to burn Qurans on the anniversary of the September 11 attacks had the right to follow through with his intentions, Supreme Court Justice Stephen Breyer said Wednesday.
In an interview Wednesday on CNN's "Larry King Live," the associate justice compared the act of burning Islam's holy book to setting the American on flag on fire when asked whether the Rev. Terry Jones had the right to carry out the controversial plan.
"We protect expression that we hate," he said. "When you have a country of 300 million different people who think different things, it is helpful. It is helpful to tell everyone, 'You can think what you want.'"
Breyer, who served in the Army, acknowledged that witnessing an American flag being burned during the Vietnam War era triggered "a physical reaction of repulsion" in him. But, he added, free speech applies equally to popular and unpopular statements.
"It's so often I hear people say -- and particularly in college students -- 'Well, that's just so terrible what he's saying.' I say, 'Oh, you think that free speech is only for people who don't say things that are terrible?'" he said.
Breyer, who was appointed by President Bill Clinton, also discussed his new book, "Making our Democracy Work," which he said is intended as a lesson on how the court works for a generation of people who haven't had the same education in civics that he did.
Yet he said he was not bothered by a poll that found two-thirds of respondents couldn't name a single Supreme Court justice, and recalled a similar poll a few years back that there were more people who could name the Three Stooges than three Supreme Court Justices.
"I say that's fine, if they don't know my name. What isn't fine at all is they can't name the three branches of government," he said. "Then we're in trouble."
Reflecting on difficult cases from his 16 years on the court, Breyer recalled two cases that found him in the minority.
Bush v. Gore was "stressful" at the time, he said, due to its magnitude and the short length of time the justices had to decide it.
"A lot turned on the outcome, we feared or thought. And what I did, like the others, is the job of the court. ... decide and control your stress. And keep yourself under control."
He also recalled tests of public school affirmative action plans in Seattle, Washington, and Louisville, Kentucky, where local officials sought to use race as a criterion to grant white students admission to the schools to integrate them.
"I thought yes, let local cities work it out for themselves. The court ruled it was contrary to the Constitution, the Equal Protection Clause," he said.
Does Breyer ever prejudge when he hears a case? "Yes. A lot," he told King.
"Think of how you make decisions. What is it to be open-minded? It isn't to be a blank slate," he said.
"You start with something. But if your mind is open, you're open to being persuaded to the contrary."