"It's a crisis in this state, and it is a crisis nationwide," Cuomo told reporters, referring to a spate of police-involved deaths that has prompted demonstrations throughout the nation.
The order also empowers the special prosecutor to review cases where questions arise over whether a civilian was armed and dangerous at the time of their death, according to a statement from the governor's office. The order will be in effect for one year.
"Without trust, the justice system doesn't work," Cuomo told CNN.
The governor's statement included words of praise for the move from the mothers and relatives of men who died at the hands of police -- as well as a range of civic and political leaders and celebrities.
One was Gwen Carr, the mother of Eric Garner, whose name appeared alongside several other relatives of unarmed civilians killed by officers.
"Nothing will bring back the lives of our loved ones, and this was never simply about our families -- it was about all those who come after us because we so deeply understand the pain and heartache of losing a loved one and then having their life not matter within our justice system," the statement said.
"While New York takes national leadership with this reform, there remains much work to be done to ensure our children and family members are no longer unjustly killed by police in the first place."
Cuomo's announcement comes nearly one year after Garner died after an officer,
Daniel Pantaleo, allegedly put him in a chokehold. Police accused Garner of illegally selling cigarettes on the street. His death was later ruled a homicide. Protesters poured onto the streets in New York and other cities when a grand jury decided not to indict the officer.
Patrick Lynch, head of the Patrolmen's Benevolent Association, which represents the city's police officers, was critical of Cuomo's order.
"Given the many levels of oversight that already exist, both internally in the NYPD and externally in many forms, the appointment of a special prosecutor is unnecessary," he said in a statement.
Lynch said the union is concerned "that there will be pressure on a special prosecutor to indict an officer for the sake of public perception and that does not serve the ends of justice."
Attorney General Eric Schneiderman, who will oversee the special prosecutor, announced the formation of the new Special Investigations and Prosecutions Unit and appointed Executive Deputy Attorney General Alvin Bragg to lead it. Bragg is a veteran prosecutor and former Assistant U.S. Attorney, according to the governor's statement.
"I can imagine no more important responsibility," Schneiderman said in a statement.
Cuomo told CNN that he's seen a growing number of incidents in New York and across the country where mainly minority communities have felt that police have been unfair with them. He said many residents believe the local district attorneys work too closely with the police, leading to a perceived conflict of interest.
The mere perception of a conflict was enough to hurt the credibility of the justice system, he said.