The Times says it interviewed multiple women about their alleged encounters with Toback over several decades; the Times says most of the women spoke on the record.
The article reported that according to the women: "In a hotel room, a movie trailer, a public park, meetings framed as interviews or auditions quickly turned sexual."
CNN has not independently verified all of the allegations, but is reaching out to the women.
Toback's agent, Jeff Berg, said Sunday he would pass a request for a comment on to his client. Berg has since ceased representing Toback. The director, 72, denied the allegations when contacted by the Times, the newspaper said.
Toback told the paper he had never met any of the women -- or if he did meet them, it "was for five minutes and (he had) no recollection." He said that for the last 22 years, it was "biologically impossible" for him to take part in the behavior the women described in the article, saying he had diabetes and a heart condition requiring medication, the Los Angeles Times reported. He declined to elaborate.
The allegations against Toback come in the wake of a New York Times story
this month detailing numerous accusations of sexual harassment against powerful movie producer Harvey Weinstein
. The New York Times identified several of Weinstein's accusers.
The allegations against Weinstein sparked the Twitter hashtag #metoo, in which women have shared personal stories of sexual harassment and abuse.
More than 40 women
have accused Weinstein
of inappropriate behavior, ranging from sexual harassment to rape. Through a spokeswoman, Weinstein has "unequivocally denied" all allegations of rape.
On Sunday, three of Toback's accusers -- Sari Kamin, Starr Rinaldi and Terri Conn -- appeared on NBC's "Nightly News."
Kamin, 37, later told CNN that she met Toback in 2003 while making copies of a script at a print shop in New York.
She said he talked about his movies. They went to dinners together, and she thought she was " being discovered" and did not want to ruin her chances.
He later coerced her to join him in his hotel room and told her to take off her clothes, she said. While she was sitting in a chair, he started rubbing his groin against her, she said.
"That's when I asked him, 'Are you trying to get off?' and he said, 'Absolutely,' " she said.
She said for a long time, she felt a sense of shame and blamed herself.
Another woman, Ashley McQueen, told CNN she was a flight attendant walking through the Los Angeles airport in 1998 when he stopped her.
"The whole thing was so surreal. I was in the middle of my trip and actually out in the airport looking for Beanie Babies," she said, referring to the line of stuffed animal characters.
She said they talked about his movie, "Harvard Man," and he gave her "homework" to watch his films. They met several times later, ran errands together and even went to a film location together, she said.
On one occasion they were in his hotel room and he told her, "I don't have a lot of time to get to know you, I need to get to know your soul so for now, I need you to get out of your clothes. I need to get to know you through my fingers," she said.
None of the women contacted the police at the time of the alleged incidents, the Los Angeles Times reported.
According to his biography on IMDb, Toback
has 17 screenplay/writing credits, and 12 credits for directing.
His first film, "The Gambler" in 1974, starred James Caan. Toback is also credited with writing and directing "Fingers" in 1978, starring Harvey Keitel and "The Pick-up Artist" in 1987 starring Robert Downey Jr., and Dennis Hopper.
In 1992, Toback was nominated for an Academy Award for his role as the screenwriter of "Bugsy" starring Warren Beatty. Toback would later direct Downey in two other films -- "Two Girls and a Guy" in 1997 and "Black & White" in 1999, according to IMDb.
Toback most recently wrote and directed "The Private Life of a Modern Woman"
starring Sienna Miller, Alec Baldwin and Charles Grodin, according to IMDb.
This story has been updated to add that Jeff Berg no longer represents James Toback.