Sarkozy came in third in the field of seven vying for the nomination for France's center-right party.
The shock winner was Francois Fillon, who staged a remarkable late surge to win the vote.
He'll contest Alain Juppe in a second-round vote for the party's nomination on November 27.
It's assumed the winner will be the one to contest the leader of the far-right National Front, Marine Le Pen, for the presidency.
Le Pen's anti-Europe and anti-immigration stance has been gaining popularity among French voters, and in an interview with CNN last week
, she said she'd been emboldened by Donald Trump's recent victory in the United States.
It "makes the French realize that what the people want, they can get, if they mobilize themselves," she said.
With more than three-quarters of polling places reporting, Fillon had around 44% of the vote followed by Juppe with about 28%.
Sarkozy brought in 20% and said in a concession tweet, "I wish the best for my country and for the one who will have to lead the France that I love -- NS"
In a concession speech, Sarkozy said he will support the 62-year-old Fillon in the runoff.
Who is Francois Fillon?
Fillon, 62, is a lawyer who served as Sarkozy's prime minister between 2007 and 2012.
As a Catholic from Le Mans, a village in north-west France, he symbolizes the traditional provincial right. Fillon aims to end the 35-hour work week, cut public spending, abolish the wealth tax, reduce immigration and invest 12 billion euros in security, defense and justice.
Just a few weeks ago, Fillon was considered an unlikely bet for the presidency but won over voters with a polished performance in televised debates.
He apparently also struck the right tone on the fight against Islamist terrorism and ISIS after publishing a new book called "Beating Islamic Totalitarianism."
After his win, Fillon tweeted a message to the French electorate.
"I thank the French people who supported me and I send a message of respect and coming together to the other voters."
Who is Alain Juppe?
Juppe, 71, is the current mayor of Bordeaux who also served as prime minister from 1995 to 1997 under President Jacques Chirac.
He's considered a moderate who promises to promote a "happy national identity" if he comes to power.
Juppe has also promised economic reform, effective border control within and outside Europe, an end to the 35-hour work week, and more investment in security and the fight against terrorism.
Because of the unpopularity of President Francois Hollande's socialist party, the Republican nominee will likely face Le Pen in the final round of the presidential race in May.
Leader of the National Front since 2011, Le Pen is a fierce euroskeptic who has said there would be no place for multiculturalism if she took power.