Kaine had said: "When Donald Trump says women should be punished, that Mexicans are rapists and criminals, he is showing you who he is."
"Look, there are criminal aliens in this country, Tim, who have come into this country illegally, who are perpetrating violence and taking American lives," Pence added.
Kaine was referencing past remarks made by Trump last July when during his campaign announcement speech, Trump said some Mexicans are criminals and are "rapists."
"When Mexico sends its people, they're not sending their best. They're not sending you," Trump said last June. "They're sending people that have lots of problems, and they're bringing those problems with us. They're bringing drugs. They're bringing crime. They're rapists. And some, I assume, are good people."
Trump also faced extensive criticism for his accusations that federal Judge Gonzalo Curiel was biased against him because he is "Mexican" -- even though Curiel was born in Indiana.
The internet reaction to Pence's remark was swift: Soon after he made it, someone created a website, www.thatmexicanthing.org, and redirected it to the nonprofit Voto Latino Action Network, which promotes Latino voices in politics.
Actress Kerry Washington tweeted after the presidential debate, "THIS! #ThatMexicanThing YES!!!!!!!!!!!!" in response to the website redirect.
Other Latinos expressed disappointment with Pence's quip, firing back on Twitter about his comment during the debate.
"Latino Twitter not happy w/ Pence. Pissed as hell w/ 'You whipped out that Mexican thing' comment. Metio la pata. (Stuck his foot in mouth)," tweeted conservative Ana Navarro, a CNN political commentator.
Department of Housing and Urban Development Secretary Julián Castro tweeted: "Pence: 'Senator, you've whipped out that Mexican thing again?' Huh? #VPDebate"