The residences under the mandatory evacuation are near a creek that appears to have been breached, Mayor Sam Liccardo said. Tens of thousands of more homes are under voluntary evacuation notifications.
One resident slept in his car instead of going to a shelter.
"We woke up at 1:30 in the morning, cops knocked on the door of our apartment. My wife got the dog out of there and I parked in the parking lot. Yeah, I slept in the car," Santos Aguilar told CNN affiliate KGO.
Another resident said the water rose fast.
"I know we lost a lot of photos. I had a lot of my clothes down there from the winter that I like to store. I know I had a lot of shoes. I had all my baby's toys, all of his stuff down there," Gabriela Martinez told the San Francisco-based station.
The area was benefiting from clear skies.
"It's safe to say none of us have been happier to see sunshine than today," Liccardo said. "The water is receding, but we are far from out of this."
There have been no civilian injuries reported as water reached several feet on some streets and crews in boats rescued more than 400 people. One firefighter was injured while gathering a family's pets.
"This is not about rain, this is about Anderson Dam," Liccardo said. "Obviously, the rain contributed to the overflowing of Anderson Dam but what we have is truly unique. We have evacuation areas that exceeded the 100-year flood zone. This is not a flood we would typically anticipate. What happened at Anderson Dam is the cause."
Anderson Dam, just southeast of the city, is at 103% of capacity and the water level at the reservoir is still 2.2 feet above the top of a spillway where water flows out of the reservoir.
City officials are looking into why Coyote Creek couldn't handle the additional water. The creek hit a historic crest of 13.6 feet on Tuesday. The flood stage is anything above 10 feet.
Liccardo urged residents in the mandatory evacuation area to stay away because the water is contaminated with fuel, oil and, possibly, sewage. Drinking water is safe, officials said on an incident management website.
He said officials will handle future storm events differently, but he said the city needs to respond to those in need before assessing what changes to make to disaster plans.
Liccardo was asked by reporters whether there had been enough warning. "If the first time that a resident is aware that they need to get out of a home is when they see a firefighter or a boat, clearly there's been a failure," he said.
More rain is in the forecast for Saturday, the National Weather Service said.
On Tuesday, residents scrambled to fortify their homes with sandbags, but it was too late for some people. The flooding had submerged some southern San Jose homes.
Residents watched helplessly as the murky water poured into their homes and one flood victim, Juanita Wilson saw her belongings starting to float around her apartment.
"That's like a nightmare," she told CNN affiliate KPIX
. "It's kind of like something you can't believe is happening, but it is happening."