Colombo, Sri Lanka(CNN) -- The United Nations office in Sri Lanka remained closed Wednesday after a three-hour siege of its staff by a government minister and his supporters led to scuffles a day earlier.
However, Cabinet Minister Wimal Weerawansa warned Wednesday that if U.N. Secretary General Ban Ki-moon did not withdraw his three-member panel by the end of the business day, he would "fast unto death, saying, "We don't know how the public would react thereafter."
Ban has appointed a three-member panel comprising an Indonesian, a South African and an American to advise him on violation of human rights and related issues when Tamil Tiger rebels were militarily defeated in May last year.
The United Nations has been concerned about accountability issues related to the military defeat of the rebels, including alleged war crimes by troops and rebels -- allegations that both parties deny.
"Ban's move is intended to bring President Mahinda Rajapaksa before a war crimes tribunal. We will not allow that to happen," Weerawansa told a news conference.
Weerawansa said U.N. staffers will be allowed to go to their office without further interference.
On Wednesday morning, protesters gathered outside the U.N. compound and removed a barrier they had placed at the entrance to the office -- a move intended to show there was no more siege and that staffers could move about.
The government information department, in a statement issued, justified Tuesday's protest.
"The government of Sri Lanka dealt with the protest outside the U.N. complex in Colombo today, in compliance with both domestic as well as international obligations," it said. "At the domestic level, Sri Lanka being a democratic society, the government had to respect the entitlement to voice opinion, including through peaceful demonstrations. Accordingly, the police permitted a peaceful gathering in front of the complex."
The government statement acknowledged that the protests would continue.
"The government of Sri Lanka expects that the U.N. complex in Colombo would continue to function as normal in the days ahead," it said. "The government understands that those who are demonstrating intend to continue with their protest, until the U.N. system revisits the matter of the Panel on Sri Lanka. As the same time, the freedom of entry and exit to and from the complex for ahthorized personnel will remain constant."
The U.N. said it has registered its strong objection to the protest.
"While respecting the right of citizens to demonstrate peacefully, preventing access to U.N. offices hinders the vital work being carried out by the United Nations each day to help the people of Sri Lanka," it said.
Weerawansa led a group of over thousand protesters on Tuesday.
Besides placards urging Ban to withdraw the appointment of the panel, protestors also carried ones that personally insulted Ban. One called him a "stooge" of the American.