(CNN) -- Rick Vetter and his teen son got a pretty good look at the legal line between privacy and security last month, as they wrapped up a day trip to Charlotte, North Carolina.
After watching the NFL's Atlanta Falcons beat the Carolina Panthers, they were looking forward to a three-hour train ride back home to Raleigh when they arrived at the train station.
Walking up a ramp toward the platform, they noticed what appeared to be a uniformed Transportation Security Administration officer holding a leashed police dog.
"He just loosened the leash on the dog, and the dog came over to check me out," Vetter said. Standing on the platform above Vetter were three other officers who appeared to be wearing bullet-proof vests.
As the guard dog smelled him, Vetter -- who has two dogs of his own -- told the officer that it probably was reacting to the smell of Vetter's pets.
"The TSA officer said 'OK' or something like that. Then it was clear that the dog had done what he needed to do, and we went on up the ramp to get on the train."
"I'm sure somebody who wasn't comfortable with dogs would have found it a lot more disconcerting than I did, but I sort of didn't worry about it," said Vetter, an attorney for the Environmental Protection Agency.
The Vetters had encountered VIPR -- special TSA Visible Intermodal Prevention and Response teams that are tasked with performing random, unpredictable baggage and security checks at passenger train, subway and bus stations as well as trucking weigh stations across the nation.
TSA officials like to point out that the acronym stands for Transportation Security Administration, not the Airport Security Administration. And that's where VIPR comes in.
Born after 2004's Madrid railway bombings, VIPR suffered some embarrassing coordination struggles, transit officials say.
The program has 15 teams and is expanding to get access to 12 new teams to spot-check thousands of transportation depots across the nation.
VIPR teams conducted 3,895 operations in "surface modes" nationwide in 2010, according to the Department of Homeland Security (PDF).
The expansion comes after intelligence from Osama bin Laden's Pakistan compound revealed al Qaeda plans to target U.S. rail systems on the tenth anniversary of 9/11.
At a time when TSA airport searches are unpopular among many air travelers, civil liberties groups say VIPR's joint participation with local police in "warrantless" searches have been "flying under the radar" in violation of constitutional protections. Transit police say it helps them better guard against attacks like those that have hit Madrid, London and Moscow since 2004.
VIPR teams join local authorities for many of their operations aimed at searching passenger bags. Authorities say officers include plainclothes and uniformed team members -- some of them armed -- who arrive without telling passengers in advance.
Officers in the joint operations then randomly ask travelers for permission to search their bags for explosives. To prevent accusations of profiling, searchers choose a random number -- eight for example -- and then search the bags of every eighth passenger before they board.
Also, VIPR observers may be in the vicinity, keeping an eye out for suspicious behavior, police say.
Local and federal authorities insist the searches are not mandatory.
But passengers who refuse are not allowed on the train, forcing some travelers to make a tough decision.
"If you're going to make a choice between having somebody search you -- even though you question whether they have the right to do that -- or having to find another way home, I'd probably have let them search me," Vetter said. Luckily, he said, it didn't come to that. Officers did not ask to search his backpack.
As an attorney for the EPA, Vetter is pretty familiar with his Fourth Amendment constitutional right protecting him from "unreasonable searches."
"At the airport, everybody now understands it's part of the process," he said. "You can either choose to deal with it or not. But in a surprise situation like that, I would not have been pleased."
New Jersey Transit Police Chief Christopher Trucillo, who works regularly with VIPR teams, acknowledged that the search system isn't perfect.
Potential attackers carrying explosives who refuse searches are free to simply drive to the next station on the line and board there.
"Because of the sheer number of passengers, there's nothing that would prevent you from doing that," Trucillo said. But there also are "things behind the scenes that are not visible to the traveling public that we employ to keep our system safe."
This isn't security, Christopher Calabrese of the American Civil Liberties Union says. It's "security theater."
Such searches offer no protection to society at the cost of passengers' civil liberties and convenience, he says. "We're very troubled by the VIPR program."
A high-profile example of VIPR's growing pains, transit officials say, is a VIPR-assisted passenger screening a year ago at Amtrak's station in Savannah, Georgia.
Instead of screening passengers as they boarded trains -- which is standard security procedure -- officers were screening passengers as they were getting off trains.
Security experts know that makes no sense, because potential terrorists probably would be interested in bringing explosives onto trains, not taking them off.
When Amtrak's police chief, John O'Connor, got wind of it, he "was very clearly angry," said Trains Magazine reporter Don Phillips, who spoke to O'Connor at the time.
O'Connor and TSA officials then hammered out an agreement on how future VIPR/Amtrak operations would be conducted.
The chief now says the Savannah operation simply "didn't make a whole lot of sense, and VIPR has since realized their misunderstanding and have corrected what they do."
"We were pleased with the way O'Connor responded," said Sean Jeans-Gail of the National Association of Railroad Passengers, who fears that the searches might affect a more than century-old tradition in America's train stations.
"The TSA needs to understand that train stations are not airports, they're publicly open hubs for community activity, which often include shopping facilities. The TSA needs to be more mindful of that."
VIPR-assisted transit and rail searches haven't been challenged in court, but it's likely, civil liberties groups say. "I think expanding the program will be problematic politically and legally," said Ginger McCall of the Electronic Privacy Information Center.
Any courtroom battle over VIPR searches will pick apart the Fourth Amendment, which bars police searches unless there's "probable cause" to believe a crime has been committed.
But the TSA claims "administrative search authority" to conduct random checkpoint searches of passengers and baggage at "surface transportation venues" without probable cause, according to TSA spokeswoman Kimberley Thompson.
"... the administrative search does not require probable cause, but must further an important government need, such as preventing would-be terrorists from bringing an explosive device onto a crowded commuter train," Thompson said.
In certain very limited circumstances, especially after the 9/11 attacks, courts have ruled that Fourth Amendment rights don't apply because of the threat of terrorism.
In a 2006 case that didn't involve VIPR, subway rider Brendan MacWade and three others sued the City of New York and its police commissioner, challenging passenger bag searches at subway entrances where people were chosen at random. MacWade lost on appeal because the court ruled the searches were legal under the Fourth Amendment's "special needs doctrine."
"The courts have said that -- because airplanes are so vulnerable to hijacking and bombs -- certain privacy interests have to be sacrificed," CNN Senior Legal Analyst Jeffrey Toobin said. "I think subways are a harder call because so many people ride them and they don't fly in the sky. But because they're underground and vulnerable, the court said the searches were OK. Would it be the same at a bus station or a busy city street? The line starts to get very tough."
Few Amtrak passengers have ever refused to be searched, O'Connor said. "We've done thousands of them, and I would say less than a handful of people have chosen to seek other transportation."
The TSA says VIPR is performing the searches often at the invitation of local or state authorities. "TSA is not arbitrarily going out there now and searching passengers and trying to violate their rights," Thompson said. "We're conducting searches in conjunction with the stakeholders."
Is VIPR working? It's hard to know.
When asked if VIPR has ever directly resulted in discovered explosives or the arrest of suspected attackers, Thompson said, "Specific operational results are considered security sensitive information. Although the value of deterrence is difficult to measure directly the presence of law enforcement transportation security personnel VIPR assets increases the difficulty with which potential terrorists plan and conduct terrorist activity."
VIPR's spot-checking role, rail security experts say, saves the high cost of deploying officers at every passenger transportation depot in the nation -- which they point out would be virtually impossible.
The surprise element of a VIPR team accompanying local police at a train station or truck weigh station on the day of a planned attack might force a terrorist to cancel the attack. "This kind of unpredictability is another tool in the toolbox to manipulate and play with their minds and cause a level of tactical deterrence," Rand security analyst Brian A. Jackson said.
As Amtrak's O'Connor puts it: "You never know where those search scenes are going to show up."
Watching the truckers
Truckers driving their rigs through Tennessee last October were surprised to be caught up in a VIPR joint exercise with local law enforcement targeting five freight truck weigh stations and two bus stations, CNN affiliate WTVF reported.
"Transportation Security officers did not search the trucks," the TSA's Thompson told CNN. "Technically, if you think about a dog walking around a truck, sniffing for trace explosives, from my understanding is still technically a search. That's the only search the VIPRs were actually involved in." TSA officers, she said, were mostly handing out fliers. "They weren't out there searching people, and they weren't out there searching trucks."
The action was described as a statewide VIPR operation, and officials told WTVF that it wasn't a response to any particular threat, which troubles the American Trucking Associations.
"Adding security personnel at weigh stations in unfamiliar federal uniforms is not likely to raise the comfort level of commercial drivers entering weigh stations, unless there is a threat to the highway sector," ATA spokesman Martin Rojas said. It "doesn't seem like the best use of TSA resources unless there is information or intelligence that supports increased highway security."
What other countries are doing
Spain and Russia significantly beefed up their rail security in the years after deadly terrorist bomb attacks killed 191 in Madrid in 2004 and 40 in Moscow in 2010.
Although there are no regular baggage checks for passengers on Spanish trains, subways and buses, the nation's bullet trains do use electronic scanners. Far more undercover plainclothes agents and uniformed officers are believed to be stationed in or near major transit hubs. Madrid's metro trains use a private security patrol force and closed-circuit video surveillance.
Moscow -- with the world's second-busiest subway system -- has spent $1.6 billion over three years to strengthen its transit security, said security analyst Rick Nelson of the Center for Strategic and International Studies.
U.S. transit officials look to VIPR to augment security during tough financial times for many communities.
"We'd prefer to see a greater investment directly with the transit agencies," said Greg Hull of the American Public Transportation Association. "Failing that, having the ability to draw upon these resources that are under the control of the Department of Homeland Security certainly is a viable alternative."
For the Vetters, their father-and-son brush with train passenger searches offered a real-world life lesson.
Rick Vetter remembers that his wife recently gave him a refrigerator magnet featuring a quote from Ben Franklin: "They who can give up essential liberty to obtain a little temporary safety, deserve neither liberty nor safety."
"The VIPR operation certainly didn't amount to that," Vetter said. "They could have handled it better in the sense that they could have provided some advance warning that they're expanding this to train stations and bus stations."
The Ben Franklin magnet sprang from his son's government lessons in school, Vetter said, which led to a family discussion about tradeoffs in society.
"You've got to decide in society where you're going to draw some of those lines," Vetter said. "You, as an individual, may or may not agree where that line is drawn, but once it's drawn, you've got some choices to make."
CNN Madrid Bureau Chief Al Goodman and CNN Senior Legal Analyst Jeffrey Toobin contributed to this report.