Despite the firestorm that followed San Francisco 49ers player Kapernick's decision to protest racial injustice by refusing to stand for the national anthem during football games this year
, Smith reminded Smerconish that he had initially jumped to defend him.
"The fact is he had every right to protest and when he was talking about racial injustices and things of that nature, you certainly can understand where he was coming from. It was a quiet protest. He didn't impede anybody's ability to watch the games or come to the games or anything like that which is why I went on 'Good Morning America,' my show 'First Take' and other outlets to support him," he explained.
However, after Kaepernick revealed that he had decided not to vote, Smith said he changed his mind about the sports star.
"I thought it compromised everything that he was standing for," he said. "And more importantly I thought it was a disrespect to our ancestors, to people who have bled and fought and died for him to have the right to do that."
As far as Smith is concerned, he emphasized, Kaepernick's lack of participation in the election had "obliterated every argument he was trying to make."
"I'm not saying his arguments have no credence whatsoever" he explained.
"I'm saying that he himself compromised his own message and because of that I don't want to listen to him anymore because the number one tool that we have in America to provoke change is our power to vote. It's something that we fought for. It's something that was exacted to us in 1964/1965. How in god's name can you sit up there and justify not voting?" he said.
"I'm not letting him get a pass on that," he added. "I'm just not."
CNN reached out to the 49ers for comment on Smith's remarks Saturday, and representatives for the team pointed to comments Kaepernick made explaining his decision not to vote from November 15.
"I thought a lot of different things about the process and what I could and couldn't do," Kaepernick said at the time. "Once again, the system of oppression is what I have an issue with."