WASHINGTON, DC - MARCH 16:  U.S. Speaker of the House Paul Ryan (R-WI) answers questions during his weekly news conference at the U.S. Capitol March 16, 2017 in Washington, DC. Ryan answered a range of questions relating to the American Health Care Act.  (Photo by Win McNamee/Getty Images)
PHOTO: Win McNamee/Getty Images
WASHINGTON, DC - MARCH 16: U.S. Speaker of the House Paul Ryan (R-WI) answers questions during his weekly news conference at the U.S. Capitol March 16, 2017 in Washington, DC. Ryan answered a range of questions relating to the American Health Care Act. (Photo by Win McNamee/Getty Images)
Now playing
01:11
Source: No vote on health care bill tonight
PHOTO: CNN
Now playing
00:50
Tom Price: Individuals will not lose coverage
Ohio Gov. John Kasich ISO for State of the Union.
PHOTO: CNN
Ohio Gov. John Kasich ISO for State of the Union.
Now playing
00:54
Kasich: 'Unless it gets fixed, I'm against it'
Tom Price sotu 6/25
PHOTO: CNN
Tom Price sotu 6/25
Now playing
02:39
Price won't say if health bill cuts his taxes
PHOTO: Fox News Exclusive
Now playing
01:15
Trump admits he called GOP health bill 'mean'
PHOTO: Senate TV
Now playing
01:24
Senator: GOP making decisions 'in a back room'
trump cbs
PHOTO: "Face the Nation"/CBS
trump cbs
Now playing
02:32
Trump: GOP plan covers pre-existing conditions
US President Donald Trump delivers a statement on Syria from the Mar-a-Lago estate in West Palm Beach, Florida, on April 6, 2017.
Trump ordered a massive military strike against a Syria Thursday in retaliation for a chemical weapons attack they blame on President Bashar al-Assad. A US official said 59 precision guided missiles hit Shayrat Airfield in Syria, where Washington believes Tuesday's deadly attack was launched.
 / AFP PHOTO / JIM WATSON        (Photo credit should read JIM WATSON/AFP/Getty Images)
PHOTO: JIM WATSON/AFP/AFP/Getty Images
US President Donald Trump delivers a statement on Syria from the Mar-a-Lago estate in West Palm Beach, Florida, on April 6, 2017. Trump ordered a massive military strike against a Syria Thursday in retaliation for a chemical weapons attack they blame on President Bashar al-Assad. A US official said 59 precision guided missiles hit Shayrat Airfield in Syria, where Washington believes Tuesday's deadly attack was launched. / AFP PHOTO / JIM WATSON (Photo credit should read JIM WATSON/AFP/Getty Images)
Now playing
01:32
Trump: 'No particular rush' on health care
Donald Trump
PHOTO: CNN
Donald Trump
Now playing
01:14
President Trump guarantees health care deal
Now playing
01:35
GOP congressman: Greater risk, greater premium
Paul Ryan
PHOTO: CNN
Paul Ryan
Now playing
02:39
Race for health care votes is on, again
TRUMP VOTERS
PHOTO: CNN
TRUMP VOTERS
Now playing
02:47
Health care failure won't deter Trump voters
Republican Congressman Charlie Dent speaks during an interview at his campaign office in  Allentown, Pennsylvania on November 2, 2016.
PHOTO: EDUARDO MUNOZ ALVAREZ/AFP/Getty Images
Republican Congressman Charlie Dent speaks during an interview at his campaign office in Allentown, Pennsylvania on November 2, 2016.
Now playing
01:21
GOP Rep.: Health law failure on Freedom Caucus
Kasich pathetic health care bill bipartisanship sotu_00000000.jpg
PHOTO: CNN
Kasich pathetic health care bill bipartisanship sotu_00000000.jpg
Now playing
01:47
Kasich: GOP, Dems not working together is pathetic
Senator Bernard Sanders asks questions to Betsy DeVos during her confirmation hearing for Secretary of Education before the Senate Health, Education, Labor, and Pensions Committee on Capitol Hill January 17, 2017 in Washington, DC.
PHOTO: Brendan Smialowski/AFP/Getty Images
Senator Bernard Sanders asks questions to Betsy DeVos during her confirmation hearing for Secretary of Education before the Senate Health, Education, Labor, and Pensions Committee on Capitol Hill January 17, 2017 in Washington, DC.
Now playing
00:55
Sanders urges Trump to join him on drug bill
Paul Ryan health care bill presser_00000000.jpg
Paul Ryan health care bill presser_00000000.jpg
Now playing
01:32
Ryan: This is a disappointing day for us
PHOTO: CNN
Now playing
01:21
Ryan: Obamacare is the law of the land
WASHINGTON, DC - MARCH 09: Australian Prime Minister Julia Gillard addresses a joint meeting of the U.S. Congress from the floor of the House of Representatives at the U.S. Capitol March 9, 2011 in Washington, DC. Gillard emphasized the long and strong bond between her country and the United States. (Photo by Chip Somodevilla/Getty Images)
PHOTO: Chip Somodevilla/Getty Images/File
WASHINGTON, DC - MARCH 09: Australian Prime Minister Julia Gillard addresses a joint meeting of the U.S. Congress from the floor of the House of Representatives at the U.S. Capitol March 9, 2011 in Washington, DC. Gillard emphasized the long and strong bond between her country and the United States. (Photo by Chip Somodevilla/Getty Images)
Now playing
01:38
House Republicans pull health care bill
John Lewis health care bill speech sot_00001310.jpg
John Lewis health care bill speech sot_00001310.jpg
Now playing
02:28
Rep. Lewis' passionate speech against GOP bill
WASHINGTON, DC - MARCH 21:  U.S. President Donald Trump speaks during a bill signing ceremony in the Oval Office of the White House March 21, 2017 in Washington, DC. President Trump has signed S.442 - National Aeronautics and Space Administration Transition Authorization Act of 2017 into law.  (Photo by Alex Wong/Getty Images)
PHOTO: Alex Wong/Getty Images
WASHINGTON, DC - MARCH 21: U.S. President Donald Trump speaks during a bill signing ceremony in the Oval Office of the White House March 21, 2017 in Washington, DC. President Trump has signed S.442 - National Aeronautics and Space Administration Transition Authorization Act of 2017 into law. (Photo by Alex Wong/Getty Images)
Now playing
00:26
White House on health care bill: 'This is it'

Story highlights

The House Freedom Caucus got its start in 2015

The group is considered the most far-right flank of the Republican conference

(CNN) —  

The rag-tag rabble-rousing House Freedom Caucus may be the linchpin to repeal and replace Obamacare at this point.

Its members, however, are Public Enemy No. 1 on Capitol Hill.

The group of conservative agitators who’ve irked leadership, held the line against spending bills and rebelled against trade priorities of leaders have now helped delay a health care vote at least one day.

Lawmakers and staffers are taking out their unhappiness on the group.

Even the White House is getting frustrated, sources say, as meetings – – including one Thursday morning with President Donald Trump – have gotten them nowhere.

Rep. Bradley Byrne, an Alabama Republican and member of the Republican Study Committee, a large group of fiscal conservatives, complained Thursday morning that the Freedom Caucus continues to throw out an expanding list of demands and suggested there was little to be done to get some in that group on board and it was time to move on.

“Those members don’t change so at some point you’ve got to say there’s nothing in the world that’s going to change their minds,” Byrne said.

A senior GOP aide said the caucus realizes they have moderates at risk with nothing to show for it.

“They don’t know how to get to yes. They just don’t. And it’s putting the whole process at risk right now,” the aide said.

Byrne blasted the move to continue talks with the hard right in the Freedom Caucus while leaving the bulk of other members in the dark.

“The vast majority of us in the Republican conference have been left out of these discussions and we have no idea what’s going on, and I think that is a problem for our leadership and I think it’s growing problem for the chances of this bill,” Byrne, who is supporter of the health care bill and part of the whip operation to help pass it.

Freedom Caucus Chairman Mark Meadows, R-North Carolina, said Thursday there are “30-40” votes against the bill at this point. He also tried to downplay his power.

“I don’t have any veto power,” Meadows told reporters. “I have one vote.”

So what is the Freedom Caucus, anyway?

A brief history

The House Freedom Caucus got its start in 2015 when a small group of members came together to push for more conservative spending and policy ideals in the House. At first, there were just nine of them, but the group, which is considered the most far-right flank of the Republican conference, grew.

Who is in it?

There is no official list of members, but the group is currently led by Rep. Mark Meadows, a North Carolina Republican. Other prominent members include Ohio Rep. Jim Jordan, Idaho Rep. Raul Labrador, Alabama Rep. Mo Brooks, South Carolina Rep. Mark Sanford and Michigan Rep. Justin Amash.

What have they accomplished?

The group – when they hold the line together – can have a lot of influence.

In 2015, they pushed to defund Planned Parenthood in a must-pass bill to keep the government funded. The group’s revolt against then-House Speaker John Boehner on a host of issues left Boehner ready to retire. During the Obama years, many of the conservative bills that made it through the House never had a shot of being signed by former President Barack Obama. Now, the group could have a lot more sway.

And while the group has insisted its relationship with House Speaker Paul Ryan is more open, the opposition by much of the group to House leaders’ health care bill has created a schism once again between the group and leadership.

Why they matter now

The group has said repeatedly that it has the votes to kill House leadership’s health care bill, and members seem to have enough sway with the White House that they’ve continued negotiating with Trump even after House leaders indicated no more changes would likely be made.

The Freedom Caucus may get a repeal of Essential Health Benefits to be included in the House leadership’s bill. That would repeal the requirement in Obamacare that insurers cover 10 benefits such as maternity care and prescription drug coverage. Including it in the bill could substantially frustrate moderates who want to be a ‘yes’ but worry that their constituents would be irate if they take away the benefits.

CNN’s Deirdre Walsh and Phil Mattingly contributed to this report.