(CNN) -- What applies in real estate, may also be said of free speech -- it is all about location, location, location.
Now a First Amendment fight between the U.S. Secret Service and anti-Bush protesters has reached the nation's highest court, testing the discretion of those using their often split-second authority to shield the President.
The government told the justices Wednesday in oral arguments that the actions of agents were related to security, while the lawyer representing the demonstrators called it unconstitutional viewpoint discrimination.
At issue: legal immunity for the agents, and whether their actions violated "clearly established" law in performance of official duties.
The court was at times animated, perplexed, and exasperated during a one-hour public session, but appeared ready to reject the free-speech claims, which could end a pending, decade-old lawsuit.
Justice Stephen Breyer spoke for many of his colleagues when he said the job of protecting the nation's leader by this "special protective force" was especially important and unique.
"Everyone understands the danger. You can't run a risk," of allowing the President to be subject to harm, he said. "At the same time, no one wants a Praetorian Guard that is above the law, and we have examples of history of what happens when you do that. So everyone is looking for some kind of line that permits the protection but denies" suppression of legitimate protest.
The incident happened during a campaign stop in the final hectic weeks before the 2004 election. President George W. Bush made a last-minute decision to dine at the Jacksonville Inn in southern Oregon.
Local law enforcement had initially allowed about 250 anti-Bush protesters to stand near an alleyway on the same side of the street as the restaurant and adjacent hotel where the President and his entourage were staying. Pro-Bush supporters were across the street.
Bush wanted to eat on the outdoor patio and the Secret Service said the change initially forced agents to order local police and state police to move the anti-Bush crowd across the street.
Fifteen minutes later, those demonstrators were moved again, about two blocks further from the inn. They claim it was done solely to put them out of earshot of the dinner guests.
The pro-Bush people were allowed to stay, resulting in them being closer to Bush.
Seven anti-Bush demonstrators eventually sued, helped by the American Civil Liberties Union. Among them, Michael "Mookie" Moss, an organic farmer and local activist.
"In challenging the immunity and lack of constitutional accountability of individuals operating under the cover of government secrecy, this case continues to set precedent while addressing one of the darker corners of our legal system," he said in a blog post. "If those in power are actively or passively given permission to act without accountability, abuse of our most basic rights will continue to occur."
Many of the facts of the case are in dispute, and a trial has not been held pending resolution of the immunity questions.
In oral arguments, most of the justices appeared mildly frustrated with the lawyers appearing before them, while trying to explore the limits of when an agent's actions become impermissible.
Justice Sonia Sotomayor questioned the government's assertion the anti-Bush crowd was moved to keep anyone out of range of committing violence.
"In fact, the pro-Bush demonstrators were across the street pretty much at a diagonal to the President, and they were permitted to remain there the entire time. They had a throwing distance of a bomb or a shooting distance as well."
Justice Anthony Kennedy repeatedly pressed the Justice Department lawyer to clarify the government's position on whether and when "subjective intent" of law enforcement should be considered.
"Would you say that under your view of the case, that there is a First Amendment interest that protesters have, but that it is virtually unenforceable in the context of crowd control?" asked Kennedy. "Because it seems to me that if this complaint doesn't survive, nothing will."
Ian Gershengorn tried to articulate a nuanced position, saying some kinds of free speech claims could still be pressed, even if the individual agents prevail in this particular case.
He mentioned current lawsuits challenging to Secret Service policy restricting the kinds of signs displayed on a parade route.
That clearly irked Justice Antonin Scalia.
"I really don't understand what the government is doing here. It seems to me you want to win this case, but not too big," she said, bringing laughter. "I would think it is in the interest of the United States and the Secret Service to say there are no" broadly-applied First Amendment claims against the agents.
The lawyer for the protesters came under tough questioning as well.
Chief Justice John Roberts pointed at lawyer Steven Wilker with a blunt hypothetical.
"You're the head of the Secret Service detail. You've got to evacuate the President right away. Do you go through the anti-Bush crowd or through the pro-Bush crowd? You've got to decide right now quickly. I'm serious. You have to make a split-second decision."
Wilker looked perplexed. "I think whichever way provides the clearest egress," he said, arms outstretched.
Roberts: "No, no. They are both the same. That was one of your propositions, that there is no way to distinguish there. It's too late. You've taken too long to decide."
Roberts continued, "If we're trying to decide whether viewpoint can ever be a security justification, we have to consider all of the possible situations."
Wilker admitted he was no security expert.
"That's actually not much of an answer for lots of reasons, but the most, if you don't know, how are we supposed to know?" Sotomayor said.
While the agents may win at the high court, the justices seemed torn about adopting a particular general principle to guide future such First Amendment claims.
A ruling in the current dispute is expected by the early summer.
The case is Wood v. Moss (13-115).