Although House Speaker Paul Ryan has sidestepped questions about his own view on the issue, many House GOP members are reticent about moving forward and setting what they see as a potentially bad precedent.
Some aren't sure that Koskinen's tenure at the agency reaches the "high crimes and misdemeanors" threshold in the Constitution for impeaching a public official, since he took over after employees were found to target some conservative groups. Others believe the House Judiciary Committee's role is to examine questions about impeachment before anyone casts a vote on the House floor.
Ryan acknowledged his caucus is deeply split on the issue and the outcome is uncertain.
"You have members on both sides of this -- members on the Judiciary Committee on both sides of this issue and this is something where the conference is going to work its will," he told reporters after meeting with House GOP members on Wednesday morning.
Ryan worked out a deal with House Freedom Caucus Chairman Jim Jordan, who has been the point person lobbying for a House vote, to hold a special meeting next Thursday with all House Republicans to discuss the issue.
But some Freedom Caucus members vow that regardless of that discussion, they will force a vote on the matter this month before Congress wraps up business before the November election.
"I fully expect a vote to happen in September," Rep. Mark Meadows, R-North Carolina, said Wednesday.
Koskinen went to Capitol Hill Wednesday to plead his case with House Republicans, meeting with a small group of moderate House GOP members.
During the meeting, Koskinen noted that he wasn't at the IRS when the targeting took place and many of those involved in the activities are no longer there. He also asserted that punishing the head of an agency would have chilling effect on people like himself who leave the private sector to serve in top government positions.
One of the moderate Republicans warned fast action on the floor could backfire.
"There has to be some level of due process in a matter like this. I would be very concerned about taking an extremely serious matter like this and just throwing it on the floor for a vote," Rep Charlie Dent, R-Pennsylvania, told reporters after the session with Koskinen.
Koskinen also met with some members of the Republican Study Committee, a large group of fiscal conservatives.
Texas Rep. Bill Flores, who chairs the RSC, said Koskinen pushed for the normal process on impeachment -- going through the Judiciary Committee -- to play out, and there was a Q & A with some tough questions that "got a little contentious."
"The challenge that he faces is that there was a massive destruction of data at a point in time when Congress had a subpoena and an evidence order over at the IRS and that data disappeared during the time that it should have been protected," Flores said. "That happened under his watch."
Leaving the Capitol, Koskinen sounded positive, even after emerging from a session where members openly pressed why they shouldn't remove him from his post. "I was delighted to be invited to both groups and I was delighted that we had a good exchange of views," he told reporters.
Louisiana GOP Rep. John Fleming told reporters that conservatives want a vote and "it will be very soon" because their constituents are demanding some response.
He listed several incidents during the Obama administration such as the "Fast and Furious" probe under former Attorney General Eric Holder and former Secretary of State Hillary Clinton's use of personal email.
"They are mad about all the corruption up here -- that we are doing nothing about it, so they just say -- won't you do something about at least one of them? That's why I think this one is the lowest hanging fruit," Fleming said about singling out Koskinen.
But Flores, who before Koskinen addressed his group said he supported moving ahead with a vote, said the commissioner did lay out his belief that there is a process in place for Congress to go through before rushing for a floor vote.
"He makes some valid points on that and I need to think about that," Flores said.
If conservatives do go to the floor and bring up a so-called "privileged resolution" to force the vote, leaders can move to table it altogether or refer it to the House Judiciary Committee, which effectively punts the issue until after the election.
But if the GOP-led House passed some impeachment measure it would move to the Senate and force Senate Majority Leader Mitch McConnell to take up the controversial issue.
McConnell's main focus for the next nine weeks is holding onto his majority in the Senate and protecting those GOP senators in tough re-election contests from taking any votes that could damage their campaigns.