As of Wednesday morning 145 people were unaccounted for, according to CNA, Taiwan's official news agency,
with four buildings in the area either tilting or already collapsed.
The quake was centered in the East China Sea about 21 kilometers (13 miles) north of the city of Hualien. Light shaking was felt in the capital of Taipei, about 120 kilometers north of Hualien, according to reports sent to the US Geological Survey.
The Marshal Hotel in downtown Hualien was one of the buildings that collapsed, Jeff Lin with the Hualien Police Department said.
About 400 rescue workers scrambled to locate people, and a crane was brought in to help, footage from CNN affiliate SET TV showed. One building apparently had collapsed onto what once was the ground floor.
The injured are mainly from the Marshal Hotel, and a few of the injured were in the city buildings. One hotel employee was rescued from the basement and three others remain missing, officials said.
At least 50 Japanese tourists were among the injured, a Hualien City police dispatcher said.
At least one bridge, badly damaged by the earthquake, has been partially reopened by police to traffic, Taiwan's official news agency CNA said.
A 5.4 aftershock hit near Hualien City shortly after the larger earthquake, according to the USGS. There have been several other strong quakes in the area in the last few days.
Cellphone video from Hualien City shows a large building leaning at a dangerous angle as sirens are heard in the background. The video shows people gathering near the building and shining flashlights on windows.
Taiwan's President Tsai Ing-wen thanked first responders in a tweet.
"Thank you to our first responders for their tireless efforts in #Hualien," she said.
"Rescue operations have been continuing night & day. Right now 145 people remain unaccounted for -- we will not rest until all are found."
Flights to the Hualien area have been canceled, Hsu Kuo-Yung, the spokesperson for the Taiwanese government, said in a news conference shown on SET TV.
Hsu Kuo-Yung said the railway department hasn't given a full status report of the damage yet but he said it seems the railways have not been damaged much. Some roads have been damaged.
'Nerves are jangled'
Laura Lo, a worker at the 7-Eleven convenience store across the street from the Marshal Hotel, said the first and second floors of the hotel appeared to be severely damaged and police officers were conducting rescue operations there.
Her store also suffered broken glass from the quake, she said, and many roads in the area were closed.
An employee at the Park City Hotel down the street told CNN that he felt the quake but there was no damage at his location.
Margaret K. Lewis, a Seton Hall University Law School professor living in Taipei, said she felt prolonged swaying at her modern high-rise apartment building in Beitou District, in the northern part of the city.
"Nothing broken, and two children slept peacefully through the event. We have since felt a few mild aftershocks," Lewis said in an email. "Nerves are jangled, but otherwise all appears well. I have not been outside to look for damage, but my expectation is that my area is generally fine."