The boat capsized in the Ohio River just outside of downtown Louisville, according to Fire Chief Greg Frederick.
"We are still hopeful that we may be able to locate somebody. We do know that the debris we found last night near the New Albany boat ramp, there is a potential there could be something miles downstream," Frederick told CNN affiliate WAVE
Three children and one adult were hospitalized, but their conditions were not known. Several people aboard the boat were members of the same family, he said.
The incident occurred Saturday night near a new bridge project on Interstate 65, he said.
When fire department rescuers arrived at the scene, the 24-foot pontoon boat was pinned against a work barge in the middle of the Ohio River, Frederick said.
Two bodies were found at the accident site -- one man and one 14-year-old boy, he said. A search for the two missing adults -- a woman and a man -- and a girl resumed Sunday, Frederick said.
"The issue is if someone has a flotation device or if they were able to latch on to something that would keep them afloat there is potential were they could be further downstream. There is also the tragic issue that they may have drowned, and it would be difficult to locate them," said Frederick.
Multiple agencies from Kentucky and Indiana, including the Coast Guard, responded to the scene, Frederick said.
A police helicopter flew above the site, and so did two law enforcement drones with thermal imaging. Debris from the boat has been found.
It was not immediately clear what caused the boat to capsize.
A camera captured the accident on video, according to Frederick. It showed the boat at about 9:23 p.m. closing in on a work barge.
"We have some video of the boat in close proximity to the work barge. I believe they are calling it a staging barge being used in the bridge construction project," said Frederick. "At this point we are speculating that the boat may have been pinned against that staging barge and at some point capsized due to the current."
There's no indication that alcohol was a factor, according to the fire chief.
The search will continue Monday.