Kampala, Uganda (CNN) -- At least three men died from their wounds Wednesday after they were shot in a confrontation with Ugandan security forces at a destroyed historic site, the Ugandan government said.
The men were part of a group of Buganda tribe members who rushed to the site of the Tombs of the Buganda Kings which was destroyed by fire Tuesday night.
Exactly how many people were wounded remained unclear despite the government statement.
While the government statement listed three dead and five wounded, the Ugandan Red Cross said it had ferried 23 injured people from the site to a Mulago hospital, 11 of them with bullet wounds.
"There several other cases, whose number I can not establish, who were either carried away by their relative or friends after sustaining bullet wounds. Others were stampeded on as they ran," said Uganda Red Cross official Susan Kabwe, who was at the tombs site.
The tomb site is a UNESCO-listed cultural site of major significance to the country's largest tribe.
It was not clear what started the fire at the Tombs of the Buganda Kings, which was built in 1882 and lies near the capital, Kampala. It contained the tombs of four ancient kings of the Buganda people along with cultural relics, including the spears of past kings.
Tensions between the Buganda and the government have historically run high, in part because President Yoweri Museveni is not from the tribe. There have been claims Museveni favors his own people, a tribe in the west, and the Buganda have said they feel persecuted.
Both the central government and the tribe began separate investigations of the fire Wednesday.
"We saw a huge fire popping out of the tombs as it rained," said Jowelia Lunkuse, 49, who operates a kiosk at the edge of the site. "The flames kept increasing high in the air and we had to close and ran away, fearing it would spread to burn us."
Museveni visited the site Wednesday morning. King Ronald Mutebi arrived moments after Museveni left, along with Medard Ssegona, the Buganda kingdom's information minister.
"We are getting all sorts of testimonies from our people whom we will use in our independent investigations, but we do not rule out acts of foul play from our enemies," Ssegona said.
A tense atmosphere gripped Kampala hours later, with angry people lining the streets and armed soldiers on patrol.
The tombs were on a hilltop around the former Buganda palace, built in 1882 and converted into the royal burial ground in 1884, according to UNESCO. The main building was low and circular, topped by a high conical dome.
UNESCO called the building "a major example of an architectural achievement in organic materials, principally wood, thatch, reed, wattle and daub. The site's main significance lies, however, in its intangible values of belief, spirituality, continuity and identity."
The tombs are where the reigning king, or kabaka, frequently carried out important rituals related to the Ganda culture, the largest ethnic group in the land-locked East African country.
The tombs were built in the "finest traditions" of local architecture and palace design, UNESCO said.
The palace with its tombs "reflects technical achievements developed over many centuries," the organization said. "It is a major spiritual center for the Buganda and is the most active religious place in the kingdom."
"The UNESCO World Heritage Centre expresses its dismay for this tragic event that devastated the site and its related symbolic values," the group said in a statement. "UNESCO is ready to mobilize international experts to help the Ugandan authorities in assessing the damage and envisaging remedial actions.
"The Tombs of Buganda Kings at Kasubi were inscribed on the World Heritage List in 2001 and were recognized as a masterpiece of human creativity both in their conception and execution, bearing eloquent witness to the living cultural traditions of the Buganda."
Buganda kings have always built their palaces on strategic hills to control the major roads to the palace and find easy ways to escape in case of an invasion or rebellion.
When they died, the traditional practice was to bury each king at a separate site to establish a royal shrine for his jaw bone, which was believed to contain his spirit.
These shrines were staffed by descendants of the kings' leading chiefs, his widows, his ritual half-sister, and a medium through which the dead king was said to communicate with his successors. Many of the shrines are still maintained today.
Journalist Katherine Haywood and Samson Ntale contributed to this report.