But both sides agree that one group didn't do enough to prevent the violence as the crowds grew and tensions flared: the police.
Critics say both Charlottesville Police and Virginia State Police stood on the sidelines Saturday as skirmishes erupted between white nationalists and members of Antifa, a broad movement of left-leaning groups. The two groups confronted each other in Emancipation Park with shields and pepper spray.
It wasn't until police declared the rally an "unlawful assembly" and Gov. Terry McAuliffe declared a state of emergency that police ordered the gathering to break up and scattered the crowds throughout the city.
Heather Heyer, a 32-year-old paralegal, was killed after a young man rammed a car into a crowd of counterprotesters.
Rally organizer: police caused melee
Jason Kessler, who organized the "Unite the Right" rally, quickly blamed the police for not keeping the peace.
"The blame for today's violence lies primarily with the Charlottesville government officials and the police officers who failed to maintain law and order, protect the First Amendment rights of rally participants, and provide for their safety," Kessler said in a statement Sunday morning.
He said officers stood idly by as counterprotesters "attacked participants of the rally" before riot police broke up the crowds.
"Instead of maintaining law and order," Kessler said, "the police purposefully created the catastrophe that led to a melee in the streets of Charlottesville and the death of a counterprotester."
Richard Spencer, a white supremacist who helped to found the so-called alt-right movement, also took to Twitter to hurl criticisms at police.
"The Charlottesville and Virginia police have blood on their hands," Spencer tweeted. "They policed the peaceful, and they exacerbated a melee. Total outrage."
'It's almost as if they wanted us to fight'
Law enforcement was met with criticism from the protesters on the left, too.
Kendall Bills, who attended the counter-demonstration with her partner, says she was attacked by one of the rallygoers but police did nothing.
"In the entire hour that I was there," she said, "at no point did I see the police intervene in any violence they were witness to, including my own."
David Straughn, another counterprotester, claims he was near Heyer when she was hit by the car. Up until that point, he said, "the police did nothing."
Once the car plowed into the crowd, the police stepped in, Straughn said.
"I will give credit where credit is due, but I will say that was too little, too late," he told CNN. "If the police had acted differently in the beginning of the day -- before 1:42 p.m. -- maybe we wouldn't be talking about Heather Heyer right now. Maybe she would still be alive."
According to a tweet from the ACLU of Virginia, police said they wouldn't intervene "until given command to do so."
Another protester, Hawk Newsome, president of Black Lives Matter of Greater New York, told CNN affiliate WCAV
the police response to "Unite the Right" was different than at rallies he's attended in the past. He said the police in Charlottesville were too far away to prevent any violence.
"The police actually allowed us to square off against each other," Newsome said. "There were fights and the police were standing a block away the entire time. It's almost as if they wanted us to fight each other."
Neither Charlottesville Police nor Virginia State Police immediately responded to CNN's requests for comment. The city of Charlottesville also did not respond.
Officials defend police response
State and city officials, on the other hand, have come out in defense of the police's actions.
In a Monday press conference, Charlottesville chief of police Al Thomas Jr. told reporters that police did what they could to keep the situation under control.
"We did make attempts to keep the two sides separate," Thomas said. "However, we can't control which side someone enters the park."
"Absolutely, I have regrets. We lost three lives this weekend -- a local citizen and two fellow officers," Thomas said when asked if he had regrets about the preparation of his officers. "We certainly have regrets. It was a tragic, tragic weekend."
"I want to thank the men and women -- local, state and federal -- our law enforcement personnel who put their lives on the line yesterday to protect us," Virginia governor Terry McAuliffe said at a church service Sunday.
"Not one single shot was fired, with all these people with weapons. No property damage. They kept us safe."
McAuliffe largely placed blame for the violence at the feet of white nationalists.
On Monday, McAuliffe said in a statement that he was ordering his team "to conduct an extensive review that will include how we issue rally permits, law enforcement preparation and response, and coordination at the local, state, and federal level," in an effort to learn from this weekend's events.
Charlottesville mayor Michael Signer, speaking on CNN Monday morning, also defended law enforcement's preparation in advance of the rally.
"We had on the ground here the largest deployment of law enforcement professionals in Virginia since 9/11," Signer said. "As I understood it, almost a thousand officers were right here on the ground."
He added that it's the government's responsibility to "set the conditions to prepare so people can peaceably assemble."
Corinne Geller, spokeswoman for the Virginia State Police, echoed the mayor's remarks and told WCAV that police don't tell people where to stand at a protest.
"That's part of the privilege of having the freedom of speech and the freedom of assembly," she said.