The move came without warning as House Republicans were in a closed-door meeting to select their nominee for speaker, with McCarthy's wife and kids in the room. Boehner subsequently postponed the vote.
"I think I shocked some of you, huh?" McCarthy told reporters following the decision.
"If we're going to unite and be strong, we need a new face to do that," McCarthy said, adding that he did not want to win the race on the House floor with only enough votes to squeak by.
A source close to McCarthy told CNN the decision to drop out came down to "numbers, pure and simple," adding that "he had the votes to win the conference vote, but there just wasn't a path to 218" -- the number of votes needed to lock down the speakership on the House floor.
The uncertain future of House GOP leadership comes less than a month before Congress must take action to raise the debt ceiling to keep the U.S. from defaulting on its debt obligations -- a critical vote conservatives have in the past sought to stall in order to pull concessions from Democrats.
Senate Minority Leader Harry Reid
was quick to call on Republicans to "bring a clean debt ceiling increase to the floors of the House and Senate immediately" to avoid a credit downgrade. "Republican chaos is likely to get worse before it gets better but the economic livelihood of the American people should not be threatened as a result of Republicans' inability to govern," Reid said in a statement.
McCarthy's candidacy ran into trouble last week
after he suggested that the House's committee on Benghazi was an attempt to hurt Hillary Clinton's poll numbers.
Asked if that affected his decision, McCarthy acknowledged: "Well, that wasn't helpful."
Rep. Dana Rohrabacher, R-California, described McCarthy's move as "courageous," saying this is "exciting" for the party because there is now a "wide open" race for speaker.
"Because of his verbal blunder last week there were some of us that were very apprehensive and this going to create great unity among Republicans," Rohrabacher said.
Meanwhile, not one to miss an opportunity, GOP presidential candidate Donald Trump suggested he was partly responsible for McCarthy's failed bid, days after he suggested McCarthy wasn't tough enough for the job.
"They're giving me a lot of credit for that because I said you really need someone very, very, tough and very smart. Ya know smart goes with tough, I know tough people that aren't smart that's the worst. We need smart, we need tough, we need the whole package," Trump said at a campaign stop Thursday in Las Vegas.
The announcement immediately set off a round of speculation about who could win the job. Perennial candidates floated included Wisconsin Rep. Paul Ryan and South Carolina Rep. Trey Gowdy — both of whom ruled it out explicitly. Several House Republicans said Georgia Rep. Lynn Westmoreland was considering it, and others suggested Oregon Rep. Greg Walden, the chairman of the National Republican Congressional Committee.
"I would consider it," Walden said when asked by CNN about members floating his name for speaker. But he said he's not actively campaigning and noted that some are pushing the idea of an interim speaker. Several candidates have suggested a senior or retiring member should serve as speaker for the next 14 months and pledge to not run again. Rohrabacher suggested Texas Rep. Joe Barton or Kentucky Rep. Hal Rogers.
Boehner said in a statement he will remain in his post until a new speaker is elected, though he has yet to announce the date for the new vote.
"I'm confident we will elect a new speaker in the coming weeks. Our conference will work together to ensure we have the strongest team possible as we continue to focus on the American people's priorities," he said in a written statement.
Boehner also canceled a scheduled appearance Thursday night on "The Tonight Show with Jimmy Fallon," an NBC spokeswoman said.
Republican Study Committee Chairman Bill Flores twice dodged reporters' questions about whether he would rule out his own run, but also spoke against the idea of a caretaker.
"An interim will not give us the opportunity to cast that big bold vision that we need. Interims are caretakers, caretakers tend to do safe things," Flores said. "The electorate put us here in November of 2014 to take big steps, and we need to find the leader that will help us take those big steps. ... The other thing that happens with interim is you have people trying to run for the permanent position, and so you have all the distractions we've gone through the last two weeks. We don't need that."
Rep. Jason Chaffetz, who late last week jumped into the speaker's race, called an impromptu news conference less than an hour after Republicans began pouring out of the GOP conference meeting. The Utah Republican said he was also "absolutely stunned, surprised and shocked."
Chaffetz said he would continue to campaign for the top House post and said "we need to find somebody that our whole body can unite behind and do what were elected to do."
"I do believe it is time for a fresh start. That was the whole genesis for my campaign, but we need to have a lot more family discussion," he said. "I think we have a lot of internal fracturing that's happening. And we need to figure out a way to unite the party."
Westmoreland joked said he is thinking about it, joking that, "I'd like to talk to my wife first."
Asked why he thought he could get 218 votes he said, "I don't know that I can, but all I can say is I'm willing to try."
Ryan, the chairman of the Ways and Means Committee and someone who had been viewed as a contender for the job, immediately said he is not interested.
"While I am grateful for the encouragement I've received, I will not be a candidate. I continue to believe I can best serve the country and this conference as Chairman of the Ways and Means Committee," Ryan said in a statement.
With conservatives again floating his name, Gowdy said he will not run for speaker. Asked if he would reconsider and join the race if his GOP colleagues urged him to get in, he replied, "No."
Rep. Tim Huelskamp, a Kansas Republican and member of the conservative House Freedom Caucus that opposed McCarthy's bid, said the decision creates a "brand new race for speaker."
"I am not the one," McCarthy told the stunned Republicans in the meeting, according to Huelskamp.
Huelskamp also took shots at McCarthy, saying the majority leader was campaigning for the top post until "three hours ago" and said the lack of "advance notice" was characteristic of the "stunts" that have defined Boehner's leadership as speaker -- including his surprise resignation the day after Pope Francis addressed a joint meeting of Congress.
And just as McCarthy got a brief heads up moments before that announcement, McCarthy also gave Boehner notice shortly before Thursday's conference meeting, a Boehner aide told CNN.
Members had no indication the move was coming. "Totally stunned," Rep. Peter King, R-New York, said on CNN.
Westmoreland met with McCarthy in his office this afternoon and said he didn't expect him to endorse anyone.
"What Kevin has done is extremely selfless, and I think he's done a brave and courageous thing," said Rep. Louie Gohmert, R-Texas. "He was close to being right there and he chose to unite the conference rather than waging battles. "
Reminder of late 1990s chaos
Speaking on CNN, Rep. Walter Jones said the current unrest reminds him of the late 1990s. Newt Gingrich stepped down as speaker and Rep. Bob Livingston was selected to take over but quickly removed himself from consideration after it was revealed he had an extramarital affair.
Jones wrote an open letter
Tuesday stating any leadership candidate should quit their campaigns "if there are any misdeeds he has committed since joining Congress that will embarrass himself, the Republican Conference, and the House of Representatives if they become public."
Jones told CNN that he was looking out for the institution and not pointing fingers at anyone in particular.
"I think when a person has been a member of the Congress — which is a very sacred duty, quite frankly, in my opinion — and they are elevated to become a leader of a party -- could be either party, Republican or Democrat -- that those in leadership must be above reproach," Jones said.
"And all I was doing, not trying to single anybody out, but was to say in this makeup of office — the majority office and the speakers office -- all the members should be made to say I have nothing in my background that could be of embarrassment to the Republican conference, the House of Representatives or the American people," he added. "That's all this was about."