The problem is, they haven't specified what it would take to get them to leave.
"We will be here as long as it takes," protest spokesman Ammon Bundy told CNN by phone from inside the refuge.
"We have no intentions of using force upon anyone, (but) if force is used against us, we would defend ourselves."
Bundy, 40, is the son of Nevada rancher Cliven Bundy
, who drew national attention in 2014 after staging a standoff with federal authorities
And like his father, Bundy said he is standing up to the federal government over land rights.
"This is about taking the correct stand without harming anybody to restore the land and resources to the people so people across the country can begin thriving again," he said.
Bundy said his group is calling itself Citizens for Constitutional Freedom, and that the group's purpose is to restore and defend the U.S. Constitution.
Here's what led up to the occupation and what may happen next:
It started with a march for ranchers
Protesters gathered Saturday in Burns to denounce the five-year sentencing of Dwight and Steven Hammond -- father-and-son ranchers who were convicted of arson.
The Hammonds have said they started a fire in 2001 to reduce the growth of invasive plants and to protect their property from wildfires, CNN affiliate KTVZ-TV
reported, but that the fire got out of hand.
The father and son turned themselves in to authorities Monday to serve their sentences, Sheriff David Ward of Harney County, Oregon, told reporters.
Addressing the armed protesters at the federal building directly, Ward said: "You said you were here to help the citizens of Harney County. That help ended when a peaceful protest became an armed occupation. The Hammonds have turned themselves in. It's time for you to leave our community, go home to your families and end this peacefully."
Bundy said officials are unfairly punishing the Hammonds for refusing to sell their land. He said it's an example of the government's overreach, especially when it comes to land rights.
But according to Billy J. Williams, the acting U.S. attorney in Oregon, the Hammonds were rightfully convicted after setting fire to about 130 acres of public land in an attempt to cover up poaching.
In an opinion piece for the Burns Times Herald
, Williams wrote that the five-year sentences are the minimum for the crimes the Hammonds committed.
Then came the occupation
After the rally supporting Hammonds, some protesters broke into the Malheur National Wildlife Refuge building.
"This refuge -- it has been destructive to the people of the county and to the people of the area," Bundy said.
He said the refuge has taken over the space of 100 ranches since the early 1900s.
"They are continuing to expand the refuge at the expense of the ranchers and miners," Bundy said.
He also said Harney County, in southeastern Oregon, went from one of the state's wealthiest counties to one of the poorest.
CNN has not independently corroborated Bundy's claims.
No employees were inside the building when protesters broke in, officials said.
Bundy said his group is armed but said he would not describe it as a militia. He declined to say how many people were with him, saying that information might jeopardize "operational security."
What the protesters want
When asked what it would take for the protesters to leave, Bundy did not offer specifics.
"The people will need to be able to use the land and resources without fear as free men and women. We know it will take some time," he said.
"I would tell any federal agent that the people are enforcing their rights and expressing their rights to restore their land and resources back to the people."
Bundy did not explicitly call on authorities to commute the prison sentences for the Hammonds. But he said their case illustrates officials' "abuse" of power.
"We are not terrorists," Bundy said. "We are concerned citizens and realize we have to act if we want to pass along anything to our children."
The Hammonds keep their distance
But the Hammonds said they don't want help from Bundy's group.
"Neither Ammon Bundy nor anyone within his group/organization speak for the Hammond family," the Hammonds' attorney, W. Alan Schroeder, wrote to Harney County Sheriff David Ward.
What authorities are doing
As of early Monday morning, there was no police presence at the building. But the FBI said it is taking the lead on investigating the situation.
"The FBI is working with the Harney County Sheriff's Office, Oregon State Police and other local and state law enforcement agencies to bring a peaceful resolution to the situation at the Malheur National Wildlife Refuge," the agency's Portland office said in a statement.
"Due to safety considerations for both those inside the refuge as well as the law enforcement officers involved, we will not be releasing any specifics with regards to the law enforcement response."
How the community is reacting
The protest has prompted Harney County School District 3
to call off classes for the week, Superintendent Dr. Marilyn L. McBride said.
The federal Bureau of Land Management office in Burns is also closed until further notice, the agency said.
And even though Bundy is not calling his group a militia, others in the community are.
"I don't like the militia's methods," local resident Monica McCannon told KTVZ. "They had their rally. Now it's time for them to go home. People are afraid of them."
What might happen next
Bundy's call for supporters to join him might "turn into a bad situation," said CNN law enforcement analyst Art Roderick, a retired U.S. marshal who investigated anti-government militias.
"What's going to happen hopefully (is) ... we don't go out there with a big force, because that's what they're looking for," he said. "The last thing we need is some type of confrontation."
He said that law enforcement has learned how to handle these types of situations in which a law may have been broken but there hasn't been any eruption of violence and no threats to lives yet.
The best approach now, Roderick said, is to wait the group out and try to figure out how to bring a peaceful resolution.
A 'Y'all Qaeda' threat?
Some Twitter users decried what they say is a double standard in the public's reaction to the occupation. They said gunmen who took over the federal building aren't called terrorists because they're white.
"#YallQaeda waging #YeeHawd on America and we're still calling it a 'peaceful protest,'" John Hulsey tweeted
. "It's domestic terrorism and we need to shut it down."
But others were quick to hit back with their own accusations of double standards.
"Justified or not, it's a protest against government abuse of power. If Oregon is terrorism, then so is #BlackLivesMatter," Paul Joseph Watson wrote.
CNN national security analyst Juliette Kayyem said there's no doubt the armed protesters in Oregon are "domestic terrorists."
"Simply because they are not Muslim jihadists does not mean they are authorized to threaten or use violence to support their political cause," she wrote
in an opinion piece.
Blogger Larry Waldbillig pointed out the historical irony of the gunmen trying to take land originally settled by Native Americans.
"Very confused why white militiamen are claiming they're 'taking back' land that was stolen by their white ancestors," he tweeted