The remarks came during a heated back-and-forth with CNN's Chris Cuomo on "New Day" over the current President's unsubstantiated claims that President Barack Obama tapped the phones in Trump Tower during the campaign.
"You have to look at this and go: Does the intelligence community under Barack Obama's presidency come at this with clean hands?" the Wisconsin Republican said.
Obama, through a spokesman, has denied Trump's wiretap claim, as has former Director of National Intelligence James Clapper.
House Speaker Paul Ryan, Senate Intelligence Committee Chairman Richard Burr and the panel's ranking Democrat, Mark Warner, said Thursday
that they've seen no evidence of Trump's accusation that his predecessor wiretapped his phones last year.
Intelligence agents have said that before Trump entered the White House, his former national security adviser, Michael Flynn, was in communication with Russia's ambassador to the United States, Sergey Kislyak. Flynn was eventually forced to resign from the Trump administration after the information was made public.
"And the contents of that conversation was released for what purpose, Chris? It was released to go after Donald Trump from the intelligence community, and it was from the leadership of Barack Obama," Duffy said.
Cuomo argued that intelligence agents discovered Flynn's communications with the Russian ambassador while searching for information about Russia's interference in the 2016 election. The US intelligence community has said Russia interfered in the election with the goal of helping Trump.
"(J)ust to be clear, that surveillance of the Russian ambassador was Russian interference in this election," Cuomo said. "It wasn't a Trump gotcha game."
But Duffy countered that there's no proof that Flynn and Kislyak's conversation touched upon Russian meddling in the election.
"Do you have any evidence that this was a conversation between Gen. Flynn and the Russians about Russian involvement in the election? This was the national security adviser talking to the Russians," he said. "But it's never been alleged that it was about fixing the election."
"I think you're twisting the context of what's going on and the questions are legit," Cuomo responded. "But I don't think they should be used as cover for an allegation by a President that was done without proof and now a position he's sticking to despite his own party saying he should back off."
While Duffy said he has not seen any evidence to support Trump's claim that his phones at Trump Tower were tapped, he said White House press secretary Sean Spicer's remarks on Thursday that Trump stands by the claim keeps the lawmaker from criticizing the President for making the accusation.
"When Sean Spicer comes out so strongly yesterday -- you've aired that clip many times -- and Donald Trump hasn't walked his comment back, it leads me to believe they may have something," Duffy said. "So I don't want to condemn them."
"I'm willing to give President Trump and Sean Spicer a little bit of time to go show us what you have," Duffy added. "And I am encouraging them this morning, as I have other days, to put it out more quickly if they have the evidence."
Cuomo argued that the Trump administration doesn't have a history of backing off its assertions even when there is no evidence of the President's claims.
"We keep coming to you to justify these claims, because you're leaders too," Cuomo said to Duffy. "And what we ignore we empower."