(CNN) -- When Jim Whittaker became the first American to stand on top of Mount Everest 50 years ago, he was anything but elated.
Reaching Earth's highest point only 10 years after New Zealander Sir Edmund Hillary became the first to summit along with Tenzing Norgay, Whittaker said 50 mph winds were "blowing like hell," compounding the already outrageous temperature of 35 below zero.
The jetstream blasted Whittaker and Sherpa Nawang Gombu as Whittaker pounded the pick of his American flag into the ice.
But when the two men looked down from their perch, 29,028 feet above sea level, they realized summiting was not their journey's end. They still had far to go, and they'd just run out of bottled oxygen on top of the world.
"Oh, boy, we've got to get down," Whittaker thought. "Getting to the summit is half of the climb. You're working so hard to get up, you don't really think about anything else."
Whittaker's expedition members' childhoods had been filled with a passion for climbing.
Richard Pownall was bitten by the mountaineering bug in 1943 when an English teacher sent students to the library to pick out a book. After reading about climbing, Pownall got a summer job working at Grand Teton National Park in Wyoming, but that stint couldn't sate his curiosity and zeal for exploration.
Whittaker discovered his passion at 14, scaling the peaks of the Northwest U.S. and, later, while guiding people up Mount Rainier in college. After summiting Mount McKinley, the highest U.S. peak, the next natural step was "the big one," he said.
When the Americans began scaling Everest, their journey was far different from that of today's climbers. For one, they were the only team on the mountain at the time.
They also had to trek 185 miles through the sweltering Chitwa Jungle, their packs stuffed with cold-weather gear for when the elevation rose. That's about the distance from Seattle, Washington, to Portland, Oregon, though the Nepalese trek involves pathways along 18,000-foot ridges, Whittaker pointed out.
Today, climbers fly 140 miles into Nepal and trek 40 miles into base camp.
Once at Everest, it wasn't long before Whittaker's expedition experienced disaster. Two days into the climb, three men were opening a route through the Khumbu Icefall -- where descending glaciers break off into jagged, car- and house-sized chunks -- when glacier pieces collapsed around them, burying them in ice, Pownall said.
He was able to climb out, but ended his summit attempt.
Jake Breitenbach, a 27-year-old guide from Jackson, Wyoming, didn't survive. He was buried deep in the ice. His body wouldn't be recovered until much later.
"You're halfway around the world," Whittaker said. "You immediately think of your family."
They were all shaken, but Everest's fierce conditions forbade them from dwelling on it. They had to keep moving. They could mourn later.
Though the view from the peak is spectacular, the Himalayas and Nepal unfolding beneath them, Whittaker said his team was more amazed by the scenery on their way back down. After months of living in thin air, they noticed the air became thicker and softer as the oxygen increased.
At one point, they found themselves clustered, looking down at a little blade of grass coming up through the scree.
"This green, emerald green -- God, it was just incredible," Whittaker said. "There is nothing growing up above, no color -- it's all snow, ice and rock. We were in tears. We had lost Jake up on the mountain but now we were coming back into life, this beautiful, lush, gorgeous planet that supports life. A little blade of grass just stunned the whole team."
Almost 49 years after summiting Everest, Whittaker, then-83, found himself back at base camp in 2012. His son, Leif, 27, wanted to reach the so-called "Head of the Sky" for a second time. Leif Whittaker had done it without his father in 2010.
Whittaker and his son had trekked to a base camp in 2003, but they had no intention of summiting.
Whittaker said he never encouraged or discouraged his son from mountaineering, but his son discovered it for himself at age 15. After being asked so many times if he'd follow in his dad's footsteps up the face of Everest, Leif found his answer on their 2003 trip.
"It was the natural power and majesty of that place that I felt some special connection to," Leif said. "I think we all are affected by landscapes in a different way, and for me, a boy who had grown up with that idea of Mount Everest in his head, seeing Everest for the first time made me want to climb it."
Whittaker hoped to walk into base camp with his son in 2012, but a day from their destination, he caught an intestinal bug. The seasoned mountaineer who had once conquered Everest was within him, telling him to keep going, but he decided it was too dicey.
Meanwhile, Leif Whittaker faced a different danger: overcrowding. Ten people died on Everest in 2012, raising questions about how many people should receive permits to scale the perilous peak.
Leif waited for more than an hour at 28,700 feet, just below the summit, as 100 people slowly descended the tricky Hillary Step. If it hadn't been for calm winds May 26, the last window of good weather, he would have been forced back down.
"Each person climbs Everest for different reasons. The reason that I climb is because I love the mountain, I love what comes with it: the view from the top, the camaraderie of good teammates, the personal challenge of the experience, pushing past your own boundaries and growing because of that experience."
Father and son experienced the magnitude of emotion that comes with being at Everest together, and Whittaker is proud his son reached the summit twice. While the achievement continues the family legacy, Whittaker wanted his children to appreciate the life around them.
"I came back from Everest without ego because you realize how insignificant you are, just a speck in the vast universe," Whittaker said. "You dwell in the silence of the forest and the high mountains. They are the highest cathedrals in the world."