SANTA CRUZ, Bolivia (CNN) -- Pro-government peasant groups are being blamed for burning dozens of ballot boxes Sunday in Bolivia's largest state of Santa Cruz, where voters are casting their ballots in an autonomy referendum.
Workers in Santa Cruz parade behind a float with a sign that reads "Autonomy."
The violence has been described as isolated by Santa Cruz's provincial Gov. Ruben Costas, who had warned of attempts to subvert the vote.
In addition to the destroyed ballot boxes, pro-government groups blocked roads and attacked polling stations in the districts of San Julian and Yapacani, according to local media reports and video of the incidents. It's unclear what effect the incidents will have on the outcome of the referendum.
Costas said he would hold the government of President Evo Morales responsible for any acts of violence surrounding the controversial vote.
A Bolivian government minister said his government will not recognize Sunday's referendum on autonomy in Bolivia's largest state, calling it "illegal and unconstitutional."
"This is the official position of Bolivia and will absolutely not change," Minister Alfredo Rada said in an interview with a Bolivian television channel Saturday evening.
The referendum in Santa Cruz is the first of an anticipated power struggle between Morales and three other eastern states. The states are rich in oil and gas reserves and are collectively known as the "media luna" for their half-moon shape.
Voters in the region are expected to pass the autonomy referendum, defying government claims of illegitimacy.
"It has to be clear that we are not going to accept an interruption of the institutional order by the hand of the Mr. Evo Morales' presidency," Costas said.
Late Saturday evening, Bolivian Army General Luis Trigo issued a statement saying "some articles" in the referendum will threaten the security of the country. Watch a discussion of the effects of the referendum »
Referendum observers -- consisting of residents, university students and police -- fanned out across Santa Cruz in an effort to monitor Sunday's vote, according to local media.
The referendum is opposed by Morales, who has said a "si" vote would carry no legitimacy.
Morales, an ally of Venezuelan President Hugo Chavez and Bolivia's first indigenous president, called on the Organization of American States to help find solutions Friday. But he downplayed any threat of violence.
"It will be one more day, like any other," he said, adding that he was open to dialogue, but considers the autonomy vote "not unconstitutional, but anti-constitutional ... an illegal act."
In an emergency meeting Friday, the OAS responded, expressing support for Bolivia's territorial integrity.
An OAS statement called on "all political actors in Bolivia to make work together," but it also expressed concern over possible violence the vote.
"The problem of violence is a central question in Bolivian politics," OAS political envoy Dante Caputo told a special meeting of the organization's permanent council. "This is the central preoccupation."
The matter traces back to July 2006, when 72 percent of voters in the affluent state of Santa Cruz voted for autonomy, as did three of the country's eight other departments -- Pando, Beni and Tarija. All of them opposed Morales' policies of land reform, energy nationalization and a new constitution.
The document that spells out the details of the proposal is what voters will decide on Sunday, a delay some critics consider too long.
"The people have had it with the sentiment of carrying out the democratic mandate," said Carlos Dabdoub, in charge of autonomy issues at the prefecture of Santa Cruz and a supporter of the "si" vote. "The towns have shirked their responsibilities regarding the mandate of the referendum."
The crisis has underscored the division between the largely poor Indians, who make up the majority and live in the mountainous western region, and the richer inhabitants -- largely of European descent -- of the eastern lowlands, which are rich in natural gas and minerals.
The governing party had previously not rejected autonomy outright.
"Autonomy as a political model of sub-state organization is viable, is positive and has a high degree of legitimacy," said Cesar Navarro, a leader in the ruling Movement Toward Socialism party.
But that's not what Santa Cruz's leaders want to do, he said.
"They are creating a mini-state inside the Bolivian state, and that implies not a state composed of autonomies but a semi-federal state with serious intentions of separatism," he said.
Though Morales has accused the pro-autonomy movement of harboring separatist aspirations, Bolivia's ex-ambassador to the United States, Jaime Aparicio, disagreed.
"It's not a separatist movement," Aparicio said. Instead, he said, it is "more of a political response" to Morales' plans to alter the constitution in a way that would grant more power to the nation's indigenous people.
He said the government's sole option is to "enter into negotiations for consensus."
Bolivia's Foreign Minister David Choquehuanca Cespedes called the referendum "nonbinding." He said the proposal "is not compatible with the new constitution or even the constitution that is currently in place."
He said Morales was prepared to negotiate on the issues of concern to the pro-autonomy movement, including discussing new formulas that would result in "more equitable sharing" of taxes and transparent "land management," but the suggestions have not been welcomed by opposition leaders.
"They don't want dialogue," he said.
Choquehuanca called the proposals for autonomy "absolutely illegal and unconstitutional" and warned that carrying them out could cause the rule of law to "be completely inverted."
"If each region begins to create its own political conditions, its own citizenry, what can happen?"
He accused the opposition of trying to use a legitimate political instrument -- a popular referendum -- "to achieve something anti-constitutional," which he likened to a coup d'etat.
"Today, we are seeing a debate that can generate sinister consequences for our democracy," he said.