Aluthgama, Sri Lanka (CNN) -- In the areas surrounding the southwest Sri Lankan town of Aluthgama, an idyllic coastal settlement popular with tourists, Muslims and Buddhists have lived side by side peacefully for generations.
But a wave of deadly communal violence that followed a rally Sunday by hardline Buddhist nationalist monks has changed that.
"The house I owned was burnt down. My family has nowhere to go," Muhsin Shihab, a father of eight children, told CNN Tuesday.
His family, which has been sheltering at a local mosque since being displaced by the rioting, hadn't eaten for a day and a half, he said.
The rally, organized by the far-right Buddhist group Bodu Bala Sena (Buddhist Power Force), was called in response to an alleged altercation in the area between a group of young Muslims and a Buddhist monk and his driver on an important Buddhist religious holiday days earlier.
Addressing the crowd of thousands Sunday, the BBS's leader, Galagoda Aththe Gnanasara, gave an inflammatory speech.
Video footage from the event shows the orange-robed monk using derogatory terms for Muslims and, to approving roars from the crowd, vowing that if any Muslim laid a hand on a member of the Sinhalese majority -- let alone a monk -- that would "be the end" of them.
After the rally, Buddhist mobs marched through Muslim neighborhoods, torching and destroying dozens of homes and shops, witnesses told CNN.
Following consecutive nights of violence, in which local medical staff say at least four people were killed and sixteen seriously injured, those made homeless by the rioting were sheltering in the town's main mosque Tuesday, shell-shocked and fearful of what may come next.
Among them was Fasniya Fairooz, an 80-year-old grandmother of three, who was at home when the mob stormed into her house in Seenawatte, a local village comprised of Sinhalese and Muslims.
"We pleaded with the attackers not to harm us. They used abusive language," she said. "They took the Holy Quran and burnt it outside... Then they looted the house."
Her family had nowhere to go, she said.
Ahmed Rahamatulla, a father of four from Seenawatte, was also made homeless by the riots.
"I have lost all my belongings. My house was burnt down. All I own today are the clothes my children wear," he said.
"I don't know where to go from here. My children are all frightened and in a state of shock."
The surrounding area is in lockdown in the aftermath of the rioting, the country's worst communal violence in years. Soldiers on armored troop carriers watch over once bustling streets; shutters are drawn on the charred remains of arson-hit stores.
In a nearby house, U.S. citizen Rameeza Nizar, 47, found herself unexpectedly stranded in her bedridden mother's home during a visit from Washington D.C. for a family event.
"Every night has been a nightmare," she told CNN. "We have not slept for fear there would be attacks. We kept our lights switched off but remained together inside the house."
'Cycle of fear'
Ayoob Saja, a doctor at a local hospital and a Muslim, said his community was in a "cycle of fear" as a result of the violence, in which the vast majority of those treated for injuries were Muslims.
He said three of the dead were Muslims, two of whom were fatally shot during the rampage on Sunday, and another who died of his injuries Tuesday.
The fourth fatality was a Tamil who worked as a watchman on a Muslim-owned farm in the nearby town of Welipenna, and was attacked during continued violence on Monday night.
Sixteen people had been seriously injured, he said, including a young man whose leg was amputated Wednesday, while hundreds of others sustained lesser injuries. More than 80 homes were also destroyed in the rioting, he said.
While a heavy military presence has been brought in to enforce a curfew and prevent further violence, it has given little comfort to the community, he said.
"The armed forces are supporting the majority," he said, referring to the mostly Buddhist Sinhalese who account for about three-quarters of Sri Lanka's population. About 10% of the country is Muslim, according to the 2011 census.
"They are guarding the majority people who attack our people."
The group blamed for inciting the violence, the Bodu Bala Sena, has denied any responsibility.
Contacted by CNN, Gnanasara said he was unavailable to comment. But Dilantha Vithanage, the BBS's chief executive, told CNN "we categorically deny any involvement by our membership in reported attacks."
He said the earlier assault on the monk on a Buddhist holy day had upset people in the Sinhalese community.
Referring to Gnanasara's speech, he said: "It is true our priest spoke in strong words. He blessed the people after chanting verses. He preached to them to conduct themselves peacefully."
The allegations against BBS, he said, were "an attempt to bring disrespect to Buddhist clergy and Buddhism."
Tacit political approval?
Buddhist radicalism has been on the rise in Sri Lanka, much as in Myanmar, where a monk-led Buddhist nationalist movement has been blamed for drumming up deadly mob violence against minority Muslim groups.
On Thursday, a prominent moderate Buddhist monk who had campaigned for interfaith harmony was found bound and dumped on a road on the outskirts of Colombo, after apparently having been abducted and assaulted, police said.
The alleged victim, Wataraka Vijitha Thero, had previously been publicly threatened by Gnanasara during an interfaith press conference held with Muslim leaders in April, a police spokesman said. He was found with cuts to his body Thursday and had been hospitalized.
Many in Sri Lanka, including President Mahinda Rajapaksa's own political allies within government, are critical of the authorities for allowing the communal violence to occur.
Mangala Samaraweera, an opposition lawmaker for the southern Matara District, told CNN that he believed the Bodu Bala Sena has the tacit support of the Rajapaksa government, a view shared by many Sri Lankans. Rajapaksa has publicly denied any link.
The Bodu Bala Sena has largely been able to operate with impunity, with previous attacks attributed to the organization going unpunished.
Rauff Hakeem, Sri Lanka's Minister of Justice and the leader the country's largest Muslim political party, said in parliament that police had been asked to stop the rally but had failed to heed the request.
He also blamed BBS for inciting the "orgy of attacks against Muslims," and told CNN he was weighing his party's future in the government -- made up of an alliance of parties -- pending the official response to the violence.
Mohamed Aslam, the local lawmaker for Hakeem's Sri Lanka Muslim Congress, also blamed police for allowing the rally to take place, and said he had nearly been shot in the aftermath.
"Muslims and Sinhalese in this area have been leading peaceful lives helping each other. This is the first time police have allowed such a meeting, where mobs were incited, to take place," he told CNN.
"The meeting on Sunday by the Bodu Bala Sena roused religious sentiments. Mobs went on the attack. The police looked the other way."
Police: We took precautions
But police Senior Superintendent Roshan Silva, in charge of the district where the violence took place, denied any police responsibility for the violence. "We took all precautions. The allegation that we were inactive is false. We had deployed police all around."
Police said 55 arrests have been made over the violence, while probes by the criminal investigation department and Colombo crimes division look into larger questions around criminal culpability for allowing the rally to proceed.
A group of more than 300 concerned Sri Lankans, including academics, lawyers and journalists, signed an open letter condemning the BBS's "hate speech," saying they believed the violence was directly linked to the inflammatory comments by Gnanasara.
"We therefore call upon the authorities to take immediate steps to arrest and charge him for the deaths and destruction in the area," read the letter.
Returning to the country after a G77 meeting of developing nations in Bolivia, Rajapaksa visited an affected Muslim town and vowed that an "impartial inquiry would be held and those responsible punished." He made no reference to the BBS.
Many Muslim businesses in Sri Lanka's capital were shut Thursday in protest at the violence.
Iqbal Athas reported from Aluthgama and Colombo. Tim Hume reported and wrote from Hong Kong.