It is the fourth, and final, Civil War-era landmark the city has removed since late April.
The effort to remove New Orleans' monuments has been part of a nationwide debate over Confederate symbols, which some argue represent slavery and injustice and others say represent history and heritage.
"Na na na na, hey, hey, hey, goodbye!" some in the crowd cheered as the statue was lowered onto a flatbed trailer.
Earlier, with work underway, Mayor Mitch Landrieu explained the city's reasons for removing the statue and other monuments at a private address.
The historical markers "celebrate a fictional, sanitized Confederacy, ignoring the death, ignoring the enslavement, ignoring the terror that it actually stood for. And after the Civil War, these monuments were part of that terrorism as much as burning a cross on someone's lawn," Landrieu said.
In a speech about the removal of the monuments, the mayor said they were landmarks that were not a true reflection of the city.
"To literally put the Confederacy on a pedestal in our more prominent places -- in honor -- is an inaccurate recitation of our full past, is an affront to our present and it is a bad prescription for our future," said Landrieu, the city's first white mayor since 1978, who is in his final term presiding over a city that is 60% African-American.
The monuments will be stored then relocated, city officials have said.
The column on which the statue stood will remain, and the city will add a water feature.
Monument removal began in April
Streets near the city's Lee Circle -- where the Robert E. Lee monument has stood for 133 years -- were blocked off by early Friday in preparation for the dismantling that's scheduled to begin sometime before 5 p.m. CT.
The city started removing the public landmarks in late April, after the New Orleans City Council voted in 2015
to remove the four Confederate markers. Recent court rulings
cleared the way for the monuments to be removed and relocated following heated public debate and legal fights.
The issue gained momentum after the 2015 massacre of nine black parishioners in Charleston, South Carolina, by a self-described white supremacist who prosecutors said posed for photos with the Confederate battle flag.
A small crowd -- including some supporting the removal and those opposing it -- gathered outside the barricaded area throughout the morning. A few men had a heated exchange after someone took an American flag from a Lee monument supporter, CNN affiliate WVUE reported
Tempers cooled when police recovered the flag and returned it. As the day went on, the crowd grew to about 100 people, the station reported.
The Lee statue, erected in 1884 in honor of the commander of the Army of Northern Virginia, is arguably the most prominent of the four because of its location at Lee Circle, a major traffic hub at the edge of the Lower Garden District and the Central Business District.
The city's famed Uptown streetcar line wraps around the circle, and nearly all Carnival parades traverse it near the end of their route.
First removal in daylight
The statues were erected decades after the Civil War to celebrate the "lost cause of the Confederacy," a movement recognized across the South as promoting white supremacy, according to a news release from the mayor's office.
Friday's removal -- in daylight, with the timing announced a day beforehand -- contrasts with the first three, which happened in the dark of night or early morning with little notice.
The city had kept quiet about the timing of the earlier ones, citing what it said were threats that some had made toward contractors who would do the work.
A summary of the first three:
• April 24:
Contractors wearing masks and ballistic vests in light of the threats removed an obelisk commemorating the Battle of Liberty Place
. The obelisk marked a deadly fight between the Crescent City White League, a group opposed to the city's racially integrated police force, and state militia after the Civil War.
• May 11:
A 6-foot statue of Confederate President Jefferson Davis
was removed from its pedestal as dozens of people -- those opposed to the removal, as well as those backing it -- looked on. The statue, which stood for 106 years, had been atop a roughly 12-foot column and depicted Davis with his right arm outstretched with palm turned upward. It towered over the street also named after him.
The equestrian statue of Confederate Gen. Pierre Gustave Toutant Beauregard, who died in New Orleans in 1893, came down
in the early morning after standing 102 years.
The statues will be put into storage while the city looks for a suitable place to display them, the mayor has said.
Private funding raised by the city will pay for the removal of the landmarks, Landrieu's office has said.
The area that formerly housed the Jefferson Davis statue will soon feature an American flag, the city announced Thursday night.
Backlash over the removals
Backlash against removing New Orleans' Confederate monuments has been building.
The Louisiana Legislature is considering a measure that would hinder local governments from removing war memorials, including those from the Civil War. The bill would allow local governments to take down a memorial only if voters approve the action at "an election held for that purpose."
Efforts to remove Confederate statues are underway in other parts of the South, including in Charlottesville, Virginia
, where a Robert E. Lee statue is scheduled to be removed. That sparked protests, including one Saturday in which torch-carrying demonstrators were led by white nationalist Richard Spencer. The event was criticized for evoking images of the Ku Klux Klan, and a counter-protest was held on Sunday night.