(CNN) -- The Department of Justice filed a lawsuit Tuesday that seeks to stop the state of Florida from purging voters from registration rolls.
Attorney General Eric Holder said the Florida program clearly violated voter registration laws.
"We have done all that we can to try and reason with the people in Florida," Holder told a Senate committee at a hearing Tuesday in explaining why the lawsuit would be filed.
Florida's Republican governor, Rick Scott, argued earlier Tuesday that removing non-U.S. citizens from lists of registered voters was a legal necessity.
Appearing on CNN's "Starting Point," Scott said the state had identified "almost 100 individuals" who had registered to vote but weren't U.S. citizens.
"Over 50 of them have voted in our elections," Scott said. "I have an obligation to enforce the laws of our land. You don't get to vote in Florida if you're a non-U.S. citizen."
The lawsuit comes after the department began questioning the legality of the state's so-called voter purge program, which would remove names from Florida's voter rolls months before the 2012 presidential election.
Florida is considered a key battleground state in the race between President Barack Obama and certain Republican challenger Mitt Romney.
Using information from Florida's Department of Highway Safety and Motor Vehicles, the state identified more than 100,000 names of non-eligible voters that could potentially be on the lists illegally.
The Justice Department contends the Florida program improperly uses the information collected from old driver's license applications, saying the data could be outdated.
"A number of persons will subsequently have become citizens and lawfully registered to vote," the department said in a letter Monday.
Critics say the plan unfairly targets minorities and paint it as an attempt to dissuade typically Democratic voters from going to the polls.
Scott said Tuesday that claim was bogus.
"This is not a partisan issue," he said on CNN. "This is not Republican or Democrat or independent issue. This is an issue that I want, all of us want, everyone wants every U.S. citizen to go and register to vote. Participate in elections. But non-U.S. citizens shouldn't be doing that."
Florida, which itself is suing the Department of Homeland Security for access to a citizenship database, was within its legal rights in moving ahead with the purge, Scott said.
In 2008, Obama won Florida with 51% of the vote, and the state is classified as a "toss up" on CNN's Electoral Map in 2012. There are 29 electoral votes at stake.