The South Carolina Republican senator's comments come in the wake of national backlash against Trump's remarks about Mexican immigrants, in which he's called some of them "rapists" and "killers," and criticism from Clinton on Tuesday in which she said none of the Republican presidential candidates support a pathway to citizenship for undocumented immigrants.
But Graham told CNN at the Atlantic Council, a Washington-based think tank focused on international affairs, that when then-Sen. Barack Obama campaigned in 2008, he promised to pass comprehensive immigration reform if elected.
"He didn't do a damn thing," Graham said. "And Hillary Clinton never lifted a finger."
Graham cited other legislation that Obama focused on early in his presidency.
"I want to remind everybody who cares about immigration reform that Obamacare, Dodd-Frank and the stimulus went ahead of you, that he had a chance to transform the system," Graham said. "He didn't lift a finger, nor did she. So I don't need a lecture from Hillary Clinton about immigration reform."
Recalling his participation in various bipartisan "gangs" that worked on immigration reform -- "I've been in every gang you can be in in the Senate. If we had tattoos, I'd have an armful," Graham quipped -- he said he didn't recall Clinton being a part of the discussions.
"I don't remember seeing her once," Graham said. "I don't remember seeing her at the table. She never came to any of the meetings I was at. She was a public voice in 2009. I never remember her saying, 'wait a minute, President Obama, you promised to do immigration reform, honor your promise.' She didn't say a word."
As for the GOP, Graham said, "My party is in a hole with Hispanics. The first rule of politics when you're in a hole is stop digging. And somebody needs to take a shovel out of Donald Trump's hand."
"I don't need a lecture from Donald Trump or anybody else about border security," Graham added. "Every bill I've ever supported would deport felons on day one. ... The system is completely broken ... but let me tell you how you fix it: Stop talking about it and get Democrats and Republicans to work together."
As for Trump, Graham said: "Here's the mistake. You're right to point out a broken immigration system, but you're wrong to say the following: That of the 11-plus million illegal immigrants, most of them are rapists and drug dealers. Most of them are good, hardworking people, cleaning our toilets, picking the crops that we all enjoy, changing the beds and working three or four jobs in the shadows to try to keep their family afloat. Most of them have come since 1986," he said, a reference to a bill signed into law by President Ronald Reagan that addressed illegal immigration.
Growing emotional as he discussed his personal story, Graham said criminals are not welcome to the U.S., but that he understands the need to keep families together.
"As to the 11 million, I am telling you as publicly as I know how to tell you, the overwhelming majority of you are here because you left a bad place to find a better place," Graham said. "You've broken our laws and you need to get right, but I see value in you as a human being and we'll try to find a way to fix this without destroying your family. When I was 21, my mom died. When I was 22, my dad died. My sister was 13. If it wasn't for family, friends and faith, I would not be sitting here. I am not going to engage in rhetoric or policies to destroy a family that's done nothing more than try to get a better life for my political gain."
Graham then assailed Trump's rhetoric.
"When it comes to casting a shadow over all of these people, to put them in the group of being rapists and drug dealers, not only is it wrong, you're dealing -- you're digging a bigger hole," he said. "And I hope every Republican candidate would say the following: We disagree with Donald Trump in this regard, that most of the people here illegally are good, hardworking people. And if we're not willing to say that as a party, we're going to lose in 2016."