Utah Rep. Jason Chaffetz and Florida Rep. Ron DeSantis, who are both GOP members of the committee, said there are grounds to remove Koskinen based on what they called his dishonesty and betrayal of the public trust. The hearing is the result an investigation into whether the IRS improperly targeted tea party and other conservative groups.
"Commissioner Koskinen had several duties. He breached every single duty he had," said Ohio Republican Rep. Jim Jordan said.
Koskinen was invited to attend the hearing but did not. The hearing turned into a highly partisan debate with Democrats arguing that there is no ground for impeachment.
"There seems to be an anti-IRS ... environment here that makes it very difficult for me to go forward without an investigation of all that's been said," said Michigan Rep. John Conyers.
Koskinen has advised the committee that he is willing to appear in the future if they desire, IRS Spokesman Matthew Leas told CNN Tuesday.
The agency released a statement saying when the committee announced this hearing, Koskinen was returning from China.
"The committee's quick timetable left him without the time to fully prepare for Tuesday's hearing," the statement said. "Koskinen believes there is an extensive record from the congressional investigations and previous hearings on the actions taken at the IRS. Although he will not be at Tuesday's hearing, he will provide written comments to the Judiciary Committee."
Critics said Koskinen allowed back-up tapes containing potentially 24,000 emails to be destroyed while failing to inform Congress of the destruction of the tapes in a timely manner.
Republicans argued that he gave false testimony before Congress regarding the back-up tapes and refused to correct the record when given the opportunity.
The emails involved Lois Lerner, the IRS official at the center of the allegations that the agency targeted conservative groups. The Justice Department in October closed its two-year investigation into the case, declining to bring charges against anyone at the agency.
At the hearing, Conyers attempted to read a letter into the record from the Commissioner, but that was protested by California Republican Rep. Darrell Issa.
The committee did not reach a conclusion on whether to seek impeachment of Koskinen before it wrapped Tuesday.
Treasury Secretary Jacob Lew released a statement Tuesday saying that Congress should spend its time focusing on more important work instead of partisan politics.
"Having begun my career working in the House of Representatives, I have deep respect for the institution and believe it should keep its focus on the needs of the American people, not the kind of political agenda that an impeachment vote here would represent," he said.