Residents earlier this week had been asked to leave an exclusion zone that stretches 7.5 miles (12 kilometers) from the center of Mount Agung.
Close to 144,000 residents -- including tens of thousands who live outside the zone -- heeded the government's warning. Now, officials say those whose homes aren't in the zone should leave the nearly 500 temporary government shelters, where extra people have taxed limited resources.
An eruption would pose no risk to those outside the zone, according to the Indonesian National Agency for Disaster Management.
"The governor of Bali has ordered that residents from 27 villages within dangerous zone, which account for about 70,000 people, must remain displaced," a statement from the agency reads. "On the other hand, as many as 73,000 people from more than 51 villages that live outside the danger zone are safe and can go home. The return of evacuees can be done independently or assisted by the government."
Seismic activity has been detected in the volcano, and the alert level for an eruption has been raised to 4, the highest warning level on the Indonesian scale.
The official exclusion zone is based on the damage field left by the last eruption, which in 1963 left more than 1,700 people dead and about 1,700 houses on the mountain's slopes destroyed. No early warning system existed at that time.
Nearby island also evacuated
In the nearby Pacific nation of Vanuatu, the entire population of Ambae island is being moved from the path of the "increasingly active" Manaro volcano, also known as Lombenben, the government said.
The fact that Mount Agung and the Manaro volcano are showing signs of erupting around the same time is pure coincidence, said Benjamin Andrews, of the Smithsonian Institution's Global Volcanism Program.
Both volcanoes sit on the Pacific Ring of Fire
, an area of intense seismic activity that stretches 25,000 miles (40,000 kilometers) from New Zealand to South America.
Ambae island, home to 11,000 residents, hasn't been evacuated on this scale since an alert in 2005, Minister of Lands and Natural Resources Ralph Regenvanu told CNN. Most are being taken away by boat, with others being airlifted to evacuation centers on Vanuatu's Pentecost and Maewo islands.
The alert level is at 4, the second highest on the Vanuatu scale.
An aerial survey Wednesday by the New Zealand Defense Force found "huge columns of smoke, ash and volcanic rocks billowing from the crater of an erupting volcano" on Ambae island, according to an agency news release.
"Ambae volcano is in an ongoing moderate eruption state," a statement from Vanuatu Meteorology and Geohazards Department said. People in nearby villages could encounter flying rocks, volcanic gases and acid rain, the statement added.