CNN —  

A flag bearing a swastika flying outside a house in a small Colorado community was taken down after dozens of residents showed up outside the home in protest.

When a neighbor displayed a Nazi flag and a Confederate flag outside his home on Election Day, November 6, residents of Fruita, Colorado, organized a peaceful protest calling for the homeowner to take down the flags, CNN affiliate KKCO reported.

Protest organizer Joe Gibson said he hoped people expressed their anger and pain by protesting against the flags being put up.

“Really, I just want a future where my kids can grow up without Nazis and hate on our streets,” Gibson told KKCO.

But not everyone felt the same. The protest was met with another protest, with residents vouching for the family’s right to fly the flag.

Paul Delancy told the affiliate the symbol was part of history and that the family has a right to fly the flag at their home.

“I’m not even here to support flying the flag, or not flying the flag,” he said. “I’m here to protect a family in defense of their right to fly the flag.”

No one has said why the flags were put up, but James Williams, who attended the protest, said both sides came to an understanding, and the flags were eventually taken down.

“His friends stepped in to defend his right, but after having a peaceful dialogue we all came to an agreement,” Williams said. “Even though he has the right to fly any flag, it’s not OK to promote hate.”

Williams said that at the time of the protest, the homeowner was not home but his friends took the initiative to take the flags down, and he hasn’t put them back up, despite telling CNN affiliate KKCO that it was his American right to fly the flag and that he had no intention of taking it down. CNN has not been able to identify the homeowner.

“We understand that people are entitled to their own opinions, but we live in a small community and we have to respect one another,” Williams said.

Williams said he hopes the protest leaves a lesson behind for everyone to follow.

“This is not about who is right or wrong. This is what our nation needs, peaceful dialogues, where two sides can come to a mutual agreement. This is how unity happens,” Williams said.