Key had been in power since 2008, and leader of the ruling National Party since 2006.
"Ten years at the top is a long time," he told reporters
in Wellington on Monday, adding that he owed it to his family to step aside.
Key, who has dominated New Zealand politics for years, winning three terms as prime minister, said he had "given everything" to the job.
"All of this has come at quite some sacrifice for the people who are dearest to me -- my family," he added.
He said that his adult children had faced "an extraordinary level of intrusion and pressure because of their father's job."
Key and his wife Bronagh have been married since 1984. They have two children, model and DJ Max, and Stephanie, an art student.
Max is a DJ and model who has been criticized in the past for his comments on social media.
The National Party will hold a special caucus on Monday, December 12 to select a new leader, Key said, after which he will tender his resignation to the Governor General.
Key said he would support whomever was chosen, but said that his deputy prime minister and finance minister Bill English "would win the election in 2017."
English previously led the National Party to its worst-ever electoral defeat in the 2002 election.
Key said he will stay in his parliamentary seat long enough to avoid a by-election.
'Say it ain't so bro'
Opponents and supporters of Key voiced their shock at his announcement.
"John Key has served New Zealand generously and with dedication," wrote Labour Party leader Andrew Little. "I wish him and his family the best for the future."
Green Party co-leader Metiria Turei said she "fought every day against John's politics but always supported his right to be a dad (and) a husband first. I wish him and his family well."
Australian Prime Minister Malcolm Turnbull told reporters in Canberra that when he heard Key was about to resign he sent him a text message: "Say it ain't so bro."
"It will be a great lost to New Zealand and a great loss to the world," Turnbull said, according
to Sky News Australia.
Turnbull's predecessor Tony Abbott complimented
Key's "fine innings."
Move 'defies political gravity'
New Zealand's political commentators appeared to agree that Key's style of exit was to be expected of him.
Jane Patterson, political editor at state broadcaster Radio New Zealand said Key had to resign ahead of the next election if he did not plan to see out another term. "His popularity is such that people have voted for National on the basis that he is prime minister," she commented on air.
"His departure will inevitably raise questions about whether there is another reason he is leaving so abruptly - that was put to Mr Key directly, but he just repeated that he was no "career politician," and that he and his family had had enough," she wrote in an opinion piece.
Fairfax News political editor Tracy Watkins wrote
that Key's resignation "played out almost entirely to script," as he had always said it would be better to choose the manner of his departure, but that it was still a shock.
"No one will begrudge Key leaving for family reasons," she said. "But the love affair between Key and the New Zealand public has been a long one. And while the initial ardor might have cooled, most kiwis would have been surprised to find out that the marriage is over."
Speculation over the reasons for Key's resignation would continue, wrote the New Zealand Herald's political editor Audrey Young
, because the move "defies political gravity."
But Young said Key's insistence that he could not have misled the public by pretending he could have completed a fourth term was true to form.
"He has almost always been a winner in life. He wants to go out as winner on his terms," she said. She suggested that New Zealand's opposition and Key's own caucus might not share his optimism that the National Party would do as well without him.
Opposition 'off life-support'
"He will be able to say he's retired undefeated as prime minister," Hooton said. "Broadly in a policy sense, people should interpret this as stability, it is almost certain that Bill English will become prime minister based on John Key's vote."
But Hooton said the National Party would be in a more difficult position without Key helming it. "John Key has won three elections in a row, that's really been based on his talent, his unique connection with the New Zealand people," he said.
Newshub commentator Duncan Garner
went further, saying Key's resignation was "National's nightmare scenario" and the opposition Labour Party should be thanking him.
"And that's because Labour will come off eight years on life-support as of today. This is genuinely massive for the opposition. They have a much better chance now that Key, the unmovable road-block, has moved himself out of the way."