(CNN) -- "My hands are tied."
Those are words that Howard Dean Bailey never wants to hear again.
The last time he heard them: an immigration judge was ordering him out of the country.
In June 2012, just days shy of his 41st birthday, the Navy veteran was deported to Jamaica, a country he hadn't lived in since he was 17.
"It's kind of stressful," Bailey said in a phone interview, but then corrected himself. "Not kind of -- very stressful."
Bailey is one of more than 23,000 permanent legal residents who were deported in 2012. The number of veterans is unknown because the Department of Homeland Security says it doesn't keep those records and neither do other government agencies.
But the club is not an exclusive one. For instance, the group Banished Veterans, founded by a deported vet Hector Barajas, is in touch with veterans in 19 countries. He said he knows of six living in Jamaica from the Vietnam War, the Gulf War and the Iraq War.
And he estimates that as many as 30,000 former service members have been deported since 1996, while other groups put the total at several thousand.
Although thousands of former service members and their families are believed to be impacted, the matter is virtually invisible even with immigration reform a hot-button issue in Washington and veterans concerns at large receiving attention as the nation steps away from 13 years of war.
As President Barack Obama reviews related policies following heavy criticism for deporting more immigrants than any previous administration, immigration groups ask that greater discretion be given for those in the country legally -- and that military service be a priority, especially with tens of thousands of men and women leaving the post-war military and transitioning back into civilian life.
Served in Gulf War
And that brings us back to the first time Bailey heard the words, "My hands are tied."
After graduating from high school in New York, Bailey did a four-year stint as a communications specialist in the Navy, serving in the first Gulf War and traveling around the world -- to Egypt, Italy, Israel and Spain. He was one of thousands serving in the early 1990s who weren't U.S. citizens.
Currently 30,000 non-citizens serve in the military, a small fraction of the total number of active duty forces.
After being honorably discharged, Bailey ran into trouble.
As he tells it, an acquaintance from the military asked him if he could mail a package to Bailey's Virginia Beach house. When the package arrived, Bailey offered to meet his friend to hand it over.
Before he got more than a mile, police surrounded his car, confiscated the box and told him it was filled with marijuana and that they had been tracking it. Bailey said he had no idea of the package's contents. He never thought to ask what was being delivered. And he never saw his acquaintance again.
"I didn't want to call my mother," Bailey said, recalling the incident and insisting that he'd never been in trouble with the law. "I'm not doing drugs; there's no way I can go to jail."
Two years later, in 1997, he was given a court date. His lawyer advised him to plead guilty to possession of marijuana with intent to distribute to avoid risk of losing his trial and receiving a harsher sentence.
Despite having an otherwise spotless record, the sentencing judge told him that his guilty plea meant he had to sentence him to 10 years in prison under mandatory minimum sentencing guidelines.
"My hands are tied," Bailey said the judge told him.
Bailey's advocates and the Homeland Security Department confirmed the main details of his case.
He ended up serving a hair more than 15 months, but little did he know at the time that his sentence was long enough to trigger more problems down the road.
After prison, he moved home to be with his wife and 1-year-old daughter. He started his own trucking company, bought a house, stayed out of trouble and thought things were going well. So in 2005, he decided to upgrade his legal status and apply for citizenship.
He told the immigration case worker that he had been charged with a felony. She said she would be back in touch. Five years later, they said they couldn't find any record of his felony, so he provided them with the court disposition. In early 2010, he was told the felony barred him from obtaining his citizenship.
"I was very disappointed," Bailey said.
Then it got worse. Five months later, in June 2010, immigration enforcement officials showed up at his home, telling him that he was being detained because of his felony.
"If I never applied for citizenship it would have been OK," Bailey said.
Law made it easier
In 1996, Congress passed the Immigration and Nationality Act, a law that made it easier to deport people who were in the country legally. Legal permanent residents, like Bailey, can be deported if they commit a class of crimes only applicable to immigrants called "aggravated felonies" or "crimes of moral turpitude."
They are crimes that include murder and weapons trafficking but also drug possession and theft -- any crime that constitutes a sentence of one year or more.
Rick Weidman, executive director for policy and government affairs for Vietnam Veterans of America said veterans facing deportation contact his organization for help.
He said he draws a line at helping violent criminals but "to deport an individual after they put their lives on the line, 98 percent of the American people have not done that."
While the Obama administration has deported more illegal immigrants than its predecessors, it began deporting fewer legal immigrants in 2010, reaching a 10-year low in 2013 of 20,153 legal resident deportees, according to homeland security data.
Nicole Navas, a spokeswoman with Immigration and Customs Enforcement, said she couldn't "speculate" on the reason why the deportation of legal residents have decreased.
David Martin, former deputy general counsel of the Department of Homeland Security in 2009 and 2010, suggested that immigration agents were given more discretion on which people to take into deportation proceedings.
He pointed to a 2011 memo by Immigration and Customs Enforcement Director John Morton that outlined what considerations to make when determining what legal residents to take into detention, including military service, community ties, family members who are citizens and cooperation with immigration enforcement -- all things that Bailey had going for him.
"ICE respects the service and sacrifice of those in military service, and is very deliberate in its review of cases involving veterans," Navas said. "ICE exercises prosecutorial discretion for members of the armed forces who have honorably served our country on a case-by-case basis when appropriate."
But the policy change came too late for Bailey.
He had already been in custody for a year and while immigration agents now have some discretion, immigration judges still have none and cannot take any of a person's positive attributes into consideration before ordering deportation for people who have committed "aggravated felonies" -- no matter how long ago the crime was committed.
Michael Wishnie, a Yale University law professor who represents veterans going through deportation proceedings, said veterans, especially those returning from war, can be even more susceptible to deportable offenses.
"Anecdotal evidence leads me to suspect that veterans have been deported more frequently and for less serious offenses," Wishnie said, hearing about the first veteran sent through deportation proceedings in 1998.
As the two-year anniversary of his deportation approaches, Bailey's sister said his deportation date is etched into her memory.
"June 1st. I will never forget it," Janice Thomas said. "It's the same day I was coming back from my honeymoon in the Dominican Republic."
Bailey said he can't find work or even leave his house very often due to the societal stigma in Jamaica of being a deportee.
He said his family in the United States has broken apart. His teen age son has gotten into trouble with the law and his teen age daughter is severely depressed.
"Sometimes it's hard for me to talk to (my daughter) because all I want to do is cry," he said. "Everything just went downhill since I left them."
His sister and mother are working to bring Bailey back home. They have met with members of the Obama administration and with lawmakers, including Sen. Joe Manchin of West Virginia, who is interested in his case.
Alisa Wellek with the Immigrant Defense Project, who is currently representing Bailey, said Obama should address the issue of tens of thousands of legal permanent residents being deported each year.
"The Obama administration has the power to exercise discretion and provide deportation relief in cases like Howard's, and should do so without any further delay," she said.
"This is worse than death in some ways," Bailey said. "It's like being buried alive."