Cuomo had been hospitalized recently to treat a heart condition. His family said he passed away, at home, from "natural causes due to heart failure."
Cuomo was governor for three terms, from 1983 to 1995.
He was married to his wife, Matilda, for more than six decades. They had five children, including current New York Gov. Andrew Cuomo, who was sworn in for his second term Thursday, and Chris Cuomo, host of CNN's "New Day."
Asked once how he wanted to be remembered, Mario Cuomo replied: "One of the simple things I wanted to achieve is -- I want to be governor. I want to be the hardest working there ever was. And I want, when it's over -- and I figured on four years at first -- I want people to say, now, there was an honest person."
Cuomo burst onto the national political stage with his keynote address at the 1984 Democratic National Convention.
To say he was a powerful speaker would be an understatement.
A Queens native, Cuomo was born in New York City, in the apartment above his father's grocery store. After a brief shot at a career in minor league baseball, he pursued a law degree and graduated at the top of his class at St. John's University School of Law.
Although the allure of public service was strong, Cuomo's early attempts at seeking political office ended in defeat.
His first electoral success came in 1978 as running mate to former Gov. Hugh Carey. Four years later, Carey stepped aside, and Cuomo entered the race and won.
He held the governorship for three terms, winning two more handily by emphasizing lower taxes, balanced budgets, public education and affirmative action. He was a strong opponent of the death penalty.
In 1993, Cuomo passed up the opportunity to be appointed to the U.S. Supreme Court, choosing instead to run for a fourth term as governor. He lost in 1994.
Having fun in the face of defeat, Cuomo made a popular commercial for Doritos, which also starred Ann Richards, who lost the Texas governorship the same year.
'You can now sleep with the greats'
President Barack Obama praised Cuomo for his faith in God and championing of progressive values in a statement issued by the White House.
"His own story taught him that as Americans, we are bound together as one people, and our country's success rests on the success of all of us, not just a fortunate few," the President said.
Cuomo announced Bill Clinton's nomination for President at the 1992 Democratic Convention.
"It was Mario Cuomo's great gift and our good fortune that he was both a sterling orator and a passionate public servant. His life was a blessing," Clinton said in a statement.
Jon Favreau, a former speechwriter for President Barack Obama, praised Cuomo's 1984 address as one of his all-time favorite speeches.
"Used to read it constantly for inspiration," he tweeted.
Cuomo's death touched people across the political spectrum.
"Our country and our region lost a giant today with the passing of Governor Mario Cuomo," New Jersey Gov. Chris Christie said in a statement. "He was a strong, eloquent leader who loved New York and its people. As an Italian-American, he was also a role model for future generations that anything was possible through hard work and education."
The Rev. Al Sharpton remembered the former governor as "the last liberal giant of New York politics."
"He was a philosopher at heart that always saw the bigger picture. Even when we would engage in debate I felt he was playing chess while I was playing checkers," he said. "Mario, you have earned it and your place in history is secure. You can now sleep with the greats."
New York City Mayor Bill de Blasio has ordered flags flying in the city that were lowered to half staff in honor of slain NYPD officers Rafael Ramos and Wenjian Liu to remain lowered for 30 days to honor Cuomo's memory.
He commended his humanity in a statement:
"Mario Cuomo was a man of unwavering principle who possessed a compassion for humankind that was without equal."