Padilla said Wednesday that creating doubt about America's free and fair election is "taking a jackhammer to the bedrock of our democracy."
Speaking to CNN's John Berman and Kate Bolduan, Democrat Padilla said allegations of 3 to 5 million cases of voter fraud nationwide in November's election were "impossible" and expressed concern over Trump's motives.
"My concern, as elections administrator, is he's simply setting the tone for any policy changes that will go further backwards as it pertains to voting rights."
"We already have needless barriers to voter registration, needless barriers to the ballot box for eligible voters not just in California but around the country," he added.
Voter fraud taken seriously
Padilla said that calling for a massive investigation when there was no basis for one was "dangerous," and a "distraction."
Trump first made the allegations of voter fraud
in a tweet back in November. "Serious voter fraud in Virginia, New Hampshire and California -- so why isn't the media reporting on this? Serious bias -- big problem!" Trump tweeted. Padilla was critical of the President's comments then, calling his allegations absurd.
He said he took claims of voter fraud seriously and had asked the President's team to share their evidence so he could investigate the claims but "months have passed, zero proof, zero evidence."
Padilla said there were no cases of irregularities of which he has been made aware.
Later on CNN's "OutFront with Erin Burnett" he said the amount of fraud is "minuscule."
"It's frankly minuscule, and it's not just a wild guess here. We have had a request for recounts in recent years whether it's a very closely contested congressional race, or a state legislative race, or maybe local city council, or mayor's race," he told Burnett. "And whenever we get to that recount and going through the very thorough protocols, a couple, a handful, single-digit difference, maybe."
"So it's not going to make a dent in that significant margin that President Trump lost by in California," he said.
Ohio's Secretary of State Jon Husted also cast doubt on allegations of voter fraud saying he was confident there were no irregularities in Ohio.
Speaking to CNN's Carol Costello, Republican Husted said any kind of investigation should be done at a state level as the "state's run the election." He said he wished the President "would take a more constructive point of view with this discussion."