"In order to deliver on the conservative agenda we have promised the American people for eight years, we must come together to find solutions to move this country forward," the Texas Republican said in a statement. "Saying no is easy, leading is hard, but that is what we were elected to do. Leaving this caucus will allow me to be a more effective member of Congress and advocate for the people of Texas. It is time to lead."
Speaking on CNN's "New Day" Monday, Poe said the group needs to "change their concept."
"The issue was always the conservatives were not brought to the table in the Republican Party," he said. "Now we've been brought to the table."
House leaders pulled the GOP health care bill from consideration Friday after it became clear Republicans did not have enough votes to pass the legislation, in part because of near-unified opposition from the Freedom Caucus.
In the days since, the House leadership announced the Republican Party would move on from its seven-year promise to repeal and replace the Affordable Care Act, and the House Freedom Caucus has taken a large share of the blame for the defeat.
One GOP leadership aide predicted that Poe's defection, which Fox News first reported, could be just the first of the defections from the group.
"I feel like they've ostracized themselves like they haven't ever done before," the aide told CNN. "I think this could be a breaking point for the membership of the Freedom Caucus."
The Freedom Caucus didn't immediately return a request for comment Sunday.
Throughout the negotiations, the caucus' chairman, Mark Meadows, said he was trying to get to "yes" and bring his members along. But in the end, the North Carolina Republican couldn't deliver the votes even after it appeared there might be a breakthrough Wednesday.
The Trump administration had offered the Freedom Caucus a repeal of Obamacare's "essential health benefits," regulatory changes that would mean insurers no longer would have to include benefits such as maternity care or mental health coverage in their plans. But the caucus rejected the olive branch, arguing that it alone would not be enough to drive down premiums.
The Freedom Caucus continued to push to repeal popular pieces of Obamacare, such as the requirement that insurers cover people with preexisting conditions, something President Donald Trump had publicly said he wanted to keep in place.
The President felt burned by the Freedom Caucus's rejection of the compromise, a White House official said.
"It wasn't about the policy," the official told CNN. "It didn't matter what policy we made because they (the Freedom Caucus) didn't want a deal. We were captives in an internal House caucus fight. The Freedom Caucus had too much interest in killing this to send a message to Speaker Ryan: 'You have too much power.'"
On Sunday morning, Trump wasted little time publicly rebuking the conservative group.
"Democrats are smiling in D.C. that the Freedom Caucus, with the help of Club For Growth and Heritage, have saved Planned Parenthood & Ocare!," the President tweeted.
Of course, members of the caucus weren't the only Republicans opposed to the health care bill. A collection of moderates including Rep. Charlie Dent of Pennsylvania and powerful House Appropriations Chairman Rodney Frelinghuysen, of New Jersey, had also come out against the bill.
But Trump now seems to recognize the House Freedom Caucus's history as a rabble-rousing collection of leadership agitators, and being a part of a group the White House thinks brought down its first major legislative effort could be a politically vulnerable position.
"They have a history of losing members after they've gone too far and their position is indefensible to the rest of the conference," the senior GOP aide said Sunday after Poe announced his departure. "It also probably doesn't help that the President of the United States publicly rebuked them this morning minutes before two of their leaders hit the Sunday shows."