The damage was especially brutal in southern Haiti, where sustained winds of 130 mph continued to punish the impoverished island nation even as the eye of the storm crossed eastern Cuba.
The "extremely dangerous" storm has killed at least seven people, including four in Haiti's neighbor, the Dominican Republic.
Officials feared a high death toll in Haiti.
"We've already seen deaths. People who were out at sea. There are people who are missing. They are people who didn't respect the alerts. They've lost their lives," Interim Haitian President Jocelerme Privert said at a news conference.
Heavy rain throughout the day caused Haitian waterways to swell.
"The river has overflowed all around us," church pastor Louis St. Germain said. "It's terrible ... a total disaster."
St. Germain, who spoke to CNN on the phone from Les Cayes, Haiti, said the storm sheared a wall off his house and tore roofs off many buildings in the area.
And in a significant setback for emergency responders and aid relief efforts, the bridge that connected Port-au-Prince with southern Haiti collapsed.
There are reports of communications towers being affected by downed trees and officials are worried it will hamper the emergency response.
Matthew was moving across Cuba on its way to the Bahamas.
The Category 4 hurricane made landfall near Les Anglais, Haiti, around 7 a.m. ET, according to the National Hurricane Center
Up to 40 inches of rain could be dumped Haiti, which is still recovering from a devastating 2010 earthquake and a cholera outbreak.
River levels rise
Les Cayes Mayor Jean Gabriel Fortuné said Tuesday morning the storm was slamming into his city. He posted videos on social media
that showed wind whipping through trees as heavy rains pelted people on the streets.
The United Nations mission in Haiti shared a photo of people wading through water in a flooded street there.
Witnesses also reported streets flooding in the capital of Port-au-Prince. Leogane was also hit hard.
Along the US East Coast governors in four states declared states of emergency and two told residents to prepare for possible evacuations. Florida Gov. Rick Scott was particularly blunt, telling residents to prepare for the state quite possibly taking a direct hit.
Death toll rising
Seven people have died in incidents connected to Hurricane Matthew within the past week, authorities said.
Four people in the Dominican Republic died, the government announced, without providing any details about how the victims passed away.
In Haiti, Guillaume Albert Moleon, an Interior Ministry spokesman, said one fisherman died on Sunday. A second fisherman is presumed dead, but the body has not been recovered.
In St. Vincent and the Grenadines, a teenager died in a landslide as he was cleaning a drain behind his house, according to Michelle Forbes, deputy director for the National Emergency Management Office. The boy died Wednesday after storms from Matthew passed.
The hurricane could cause further devastation for Haiti as much of the country's infrastructure remains weak
after the 2010 earthquake that killed more than 200,000 people.
More than 300,000 people are in shelters across the country, the United Nations said.
After the storm clears Haiti, residents could face risks from another threat: standing water.
"That means a potential spike in cholera cases," said John Hasse, the humanitarian aid agency World Vision's national director in Haiti. "Other mosquito-borne diseases that have been more or less controlled are going to rear their heads."
Haiti is still recovering from a post-quake cholera outbreak
that killed 10,000 people.
Collision course for Cuba
Forecasters said Matthew could dump up to 20 inches of rain in Cuba.
The United States, taking no chances, airlifted 700 family members of military personnel stationed at Guantanamo Bay to Florida. The 61 detainees held by the United States as alleged enemy combatants will not be evacuated, officials added.
After skirting Cuba, Matthew is expected to hit the Bahamas and turn toward the United States. It should lose some strength, but still have 115 mph winds.