Petersen, who was up for a lifetime appointment on the US District Court for the District of Columbia, struggled to answer basic questions during an exchange with Sen. John Kennedy, a Louisiana Republican. The exchange went viral after Sen. Sheldon Whitehouse, a Rhode Island Democrat, tweeted a link to the five-minute long exchange.
In a letter to Trump, Petersen said he is "no stranger to political realities" and did not wish to be a "continued distraction" to the President's or administration's work. He bemoaned his performance at his confirmation hearing, saying, "I had hoped that my nearly two decades of public service would carry more weight than my worst two minutes on television."
Petersen, who currently serves as a commissioner on the Federal Election Commission, was peppered with questions on how many depositions he had worked on, how many jury trials he had tried and the last time he had read the Federal Rules of Civil Procedure.
His answers called into question his fitness for the job and Trump critics seized on the video as proof that the President is putting ideology ahead of qualifications.
Petersen acknowledged during the hearing that his "background was not in litigation," even though he was up for a judgeship.
"I understand the challenge that would be ahead of me if I were fortunate enough to become a district court judge," he said.
Kennedy told CNN after the hearing that he hoped Petersen's nomination would be withdrawn.
"I'm hoping the White House will pull down Mr. Petersen," Kennedy said. "I don't wanna see him suffer."
He added: "I think he's whip smart, probably. But you can't just walk into a federal courthouse for the very first time and say 'Here I am, I think I wanna be a judge.' It just doesn't work that way."
The White House last week defended Peterson.
"Mr. Petersen has spent nearly a decade as a commissioner of an important federal agency overseeing its litigation on regulatory issues -- the very kinds of issues federal district court in DC decides," Hogan Gidley, a White House spokesman, said Friday. "It is no surprise the President's opponents keep trying to distract from the record-setting success the President has had on judicial nominations, which includes a Supreme Court justice and 12 outstanding circuit judges in his first year."
The statement ignored the fact that Kennedy, a Republican whom backed Trump during his campaign, was the senator who led the charge against Petersen.