All evacuation orders will be lifted Sunday evening, Los Angeles County Deputy Fire Chief John Tripp said.
Since the inferno started Saturday, it has already scorched more than 500 acres, Tripp said.
The fire was raging on the slope of a canyon, "which makes it very dangerous for us to do a direct attack," Tripp said.
The wildfire broke out Saturday afternoon in the Calabasas area, a relatively affluent part of Los Angeles County that is home to a handful of celebrities, including Jessica Simpson and Toni Braxton, authorities said.
About 3,700 homes -- or about 5,000 people -- were ordered to evacuate in Calabasas and nearby Topanga, mountainous communities west of Los Angeles and north of Malibu.
"The fire is not on the road, it is halfway up the side of a mountain," Tripp said Sunday morning. "That's why we can't let people back in their homes."
So far, the blaze has destroyed a large building and damaged two homes, he said.
How it started
The fire started around 4:15 p.m. Saturday when a pickup truck struck a power pole on Mulholland Highway in Calabasas, the sheriff's department said.
"Witnesses reported that the truck was traveling at a high rate of speed before colliding into a power pole, causing the pole to fall and a transformer to explode, thus igniting the Calabasas fire," Deputy Jeffrey A. Gordon said.
Temperatures in nearby Topanga reached the mid-90s earlier in the day, according to the National Weather Service.
The fire has been dubbed the "Old Fire" due to its proximity to Old Topanga Canyon Road. It came right up to the edges of Calabasas High School and Viewpoint School.
Fighting blaze with garden hoses
About 400 firefighters, along with water-dropping helicopters, worked to contain the blaze, Tripp said.
He said three firefighters were injured. Two suffered knee injuries, and one had a "cardiac event."
"This is extremely arduous work for them to do," Tripp said.
Steve Gentry, a photojournalist who filmed the firefighters, told CNN the situation was "very serious."
"At one point, the fire was licking right up through to the backyards of a lot of the homes of the residents in that area, to the point where even sheriffs' deputies were grabbing garden hoses and using them to fight the fire," he said.
"Things here in California, they're very dry and it doesn't take a whole lot for a fire to get going and spread really rapidly."
Pictures of the blaze show residents staring at bright, tall flames creeping toward the sky, visible from nearby streets.
Roseann Bleiweiss' son told CNN the fire looked like "something out of a movie."
One social media user said ash could be spotted as far away as Dodger Stadium, about 30 miles away.
Celebrities also spoke about the fire.
Ellen DeGeneres tweeted that her brother-in-law was out fighting the blaze.
"Sending thanks to Portia's brother Michael & all the rescue workers on the scene of this dangerous fire," she said.
Calabasas and the nearby Santa Monica Mountains are not strangers to wildfires -- the area has experienced many dangerous blazes over the years.
A particularly strong fire in 2013
burned nearly 28,000 acres in two days, threatening thousands of homes in Ventura County.