(CNN) -- A group of indigenous Australian protesters made headlines for a second day in a row Friday after they gathered outside the Australian Parliament and set fire to the country's flag.
The move came a day after security officers dragged Prime Minister Julia Gillard out of a Canberra restaurant after scores of angry protesters surrounded the building during a luncheon ceremony.
Aborigine leaders on Friday criticized the actions of the group, saying they jeopardized efforts to reconcile indigenous Australians with the broader society.
Video on the website of the Australian Broadcasting Corp. showed a circle of protesters outside Parliament in Canberra setting light to an Australian flag and chanting, "Always was, always will be Aboriginal land."
"I think it was totally uncalled for," said Tom Calma, a co-chair of Reconciliation Australia, which promotes the improving of relationships between Aborigines and the wider Australian community. "It's just not acceptable that they burn the flag of Australia."
The police are investigating the lighting of a fire in an unauthorized place, a police spokeswoman said. She said the act of burning the Australian flag is not a criminal offense in Canberra.
The move by the protesters, connected to a long-running Aboriginal demonstration known as the "tent embassy," was likely to add to dismay across the country after the disturbance involving Gillard on Thursday, Australia's national day.
Security officers took Gillard and opposition leader Tony Abbott out of the restaurant after between 50 and 100 protesters circled it during a lunchtime ceremony, bashing windows and brandishing sticks and rocks, according to the police.
Gillard was presenting medals to emergency service workers during an event for the national day. A spontaneous protest erupted nearby among an Aboriginal rights group commemorating the 40th anniversary of the tent embassy.
Protesters were chanting "shame" and "racist" as they banged on the restaurant's three glass sides, according to the Australian Broadcasting Corp.
They were apparently upset about remarks Abbott made earlier Thursday suggesting it may be time to reconsider the tent embassy's relevance.
The makeshift embassy was set up 40 years ago by four Aboriginal men who planted a beach umbrella on the lawn in front of Parliament House in Canberra to protest the failure of the coalition government at the time, led by Prime Minister William McMahon, to recognize Aboriginal land rights. It has endured despite successive government attempts to close it.
"I think the indigenous people of Australia can be very proud of the respect in which they are held by every Australian," Abbott had said in response to a question about whether the tent embassy was still relevant, according to a transcript of the comments posted on his website. "And yes, I think a lot has changed since then and I think it probably is time to move on from that."
Local media reported that Gillard was visibly shaken, and stumbled during the encounter.
Photographs showed her being led to a waiting vehicle by a group of at least seven security officers, losing a shoe in the process.
Video showed her being hustled into the vehicle, surrounded by security officers, some carrying shields, as protesters shouted, "Shame on you."
The shoe was collected by protesters, who proclaimed it a trophy.
But Aboriginal leaders were unimpressed.
"It's pretty appalling behavior," said Mick Gooda, the commissioner for Aboriginal and Torres Strait Islander social justice at the Australian Human Rights Commission.
"And while we enjoy the right to protest and raise issues in this country, I don't think we should be resorting to that type of violence," he said Friday on Australian Broadcasting Corp. radio.
There was also criticism of the protesters' actions in the Australian press.
"Those who want to preserve the embassy should make their point in a peaceful, respectful way," said an editorial on Friday in The Australian, a national daily. "Their actions yesterday can only have damaged their cause."
There were no injuries Thursday, and no arrests were made, the police said. Video from the ABC showed some minor struggles breaking out between the authorities and protesters.
A spokesman for the tent embassy, Mark McMurtrie, said Abbott's remarks and the police's behavior had incited the disturbance.
Speaking Friday on Australian Broadcasting Corp. radio, McMurtrie said that news media coverage of the events had overly dramatized the protesters actions.
"There was no threat to them at all," he said, referring to Gillard and Abbott.
The episode took place against the backdrop of efforts to change the nation's constitution to give better recognition to indigenous Australians, often referred to as Aboriginal and Torres Strait Islander peoples.
Last week, a panel issued a government-commissioned report suggesting ways the constitution could be altered to achieve that aim.
Indigenous Australians have suffered at the hands of later settlers and the government they established. Australian politicians have since apologized for the past mistreatment, but Aborigines remain disadvantaged socially and economically compared with the overall population.
Explicit references to Aborigines in the original Constitution, drafted in the late 19th century, were subsequently deemed to be negative. Australians voted overwhelmingly to remove those points in a 1967 referendum, but many people say the document can be further improved to acknowledge the role of the country's indigenous population.
The panel -- which included Aboriginal leaders, business executives, legal experts and members of the main political parties -- has handed over its report to Gillard, whose government has promised to hold a referendum on the matter by the next general election.
The events of the past two days may have made that process more complicated for those in favor of the referendum.
"I think it's going to increase the challenge," said Calma of Reconciliation Australia. "There was always going to be a challenge to get people on board with the constitutional changes."
He said the tent embassy protesters' actions would detract from "successful advocacy taking place between indigenous groups and the government at the moment on reconciliation."
CNN's Hilary Whiteman and journalist Hugh Williams contributed to this report.