Kalam, Pakistan (CNN) -- Two weeks ago, as residents of Kalam in the Swat district of northwestern Pakistan were busy picking up the pieces of their existence after flooding had ravaged the town, a loud explosion shattered the quiet of the night.
It was 2 a.m., the local deputy mayor Amir Saeed said, and Kalam's girls' school had been bombed.
"It was the Taliban," Saeed told me as we stood by the destroyed building. "We know exactly that it was them."
"I was very scared when I heard the blast. I could not sleep after that and in the morning I saw the school was damaged," 10-year-old Nadia said as she and three other girls walked through the rubble.
No one was injured in the bombing and Pakistan's military says it was an isolated incident, but international observers and the U.S. government fear the Taliban and other Islamist groups might use the breakdown in public order caused by the floods to gain ground.
"We put so much effort into this area. There was so much progress in places like Swat and now all of that has been washed away by this flood," said Richard Holbrooke, the U.S. envoy to Afghanistan and Pakistan, while touring the devastated areas earlier this month.
The Taliban controlled much of Swat for an 18-months period that ended in the spring of last year when the Pakistani military launched a massive campaign against the militants, ousted them and restored civil order.
Now tens of thousands of Pakistani soldiers are involved in flood relief.
And Maj. Gen. Javed Iqbal Ramday, who is charge of flood relief and security in Swat, scoffs at the idea that militants could make a comeback in places like North Waziristan.
"While we are undertaking the flood relief operations and the flood relief efforts, at the same time we have a posture where we are already taking care of the threats coming back into the area," the general told CNN in an interview AT his headquarters in Khwazakhela.
Pakistan's military says it is determined to hold the ground it fought for last year and prevent militants from seeping back in.
Outposts and machine gun nests overlook the area around the town of Kalam, some of them high up in the mountains surrounding the Swat River.
A heavy military presence can also be seen on the streets with checkpoints searching those coming in and out of the area.
The speed and dynamics of the flood relief efforts could also play a role in deciding whether the population will go with the government or the militants.
The leadership in Islamabad was heavily criticized as many perceived the reaction to the disaster to be slow and indecisive.
And the general acknowledges that the military will not be able to keep the edge unless it has the local population on its side. "The people of this area are going to have to make a decision with regard to not providing them the space. And I think that decision is already known to Pakistanis, to the people of Swat and most importantly to the militants -- that they have again said a resounding no to the terrorists."
Even when the Taliban held sway in Swat, the people of Kalam resisted the militants and some even died fighting against them.
Saeed, the deputy mayor, said they would do the same now. "We are well organized to take on the Taliban, we have organized defense committees and volunteers at every village level," he said.
So far, the Taliban have had no luck retaking Swat but when I asked Saeed if he could be sure something like the bombing of the girls' school could be prevented in the future he said attacks could never be ruled out.