The city's mayor, Mike Signer had a blunt message for them: "Another despicable visit by neo-Nazi cowards. You're not welcome here! Go home!"
Charlottesville was rocked by violence nearly two months ago during clashes between white supremacists and counterprotesters. Heather Heyer, 32
, was killed in the event August 12 when a car plowed into a crowd of counterprotesters.
On Saturday around 7:40 p.m., a group of about 40 to 50 people, including Spencer, gathered at Emancipation Park, where the statue of Confederate Gen. Robert E. Lee stands, Charlottesville police said.
The group held a brief rally that police estimated took about five to 10 minutes. They dressed similarly in white shirts and some wore sunglasses in the dark while chanting, "You will not replace us. You will not erase us."
"Hello, Charlottesville. We have a message. We're back and we're going to keep coming back," one speaker said.
Spencer and his supporters alleged that Charlottesville was suppressing their speech and said that its residents should "get used to the alt-right. You're going to have to get used to white identity."
After the rally, they left the park, boarded a tour bus and departed from the city, Charlottesville police said. Police cars followed the tour bus to make sure the group was leaving the city.
"Our department is conferring with city leadership and the Commonwealth Attorney's office to determine what legal action may be taken in response to this event," police said in a statement.
Virginia Gov. Terry McAuliffe tweeted: "We are monitoring this situation as we continue to oppose these racists and their message of hate."
Spencer has now held three torch rallies in his former college town. The first event occurred in May
, also at Emancipation Park where they protested the city's plans to remove the Lee statue. The second event was in August
at the University of Virginia.