Please, Mr. President, don't leave me behind.

That's the gist of former U.S. subcontractor Alan Gross' plea to President Barack Obama in a letter on the fourth anniversary of Gross' imprisonment in a small military prison cell in Cuba.

In the letter, Gross, convicted by a Cuban court of "acts against the independence or territorial integrity of the state," says he spends 23 hours a day in a small cell with two other inmates, is in poor health and is largely cut off from the outside world.

"With the utmost respect, Mr. President, I fear that my government -- the very government I was serving when I began this nightmare -- has abandoned me," wrote Gross, a former subcontractor for the U.S. Agency for International Development. "Officials in your administration have expressed sympathy and called for my unconditional release, and I very much appreciate that. But it has not brought me home."

His wife, Judy Gross, told CNN's "The Situation Room with Wolf Blitzer" that she's fed up with apparently empty promises from government officials who have said they'll help.

"It's emotional. ... It causes me a lot of anger. I have to say right now I'm angry at the U.S. government. I'm angry at the Cuban government," she said Tuesday. "I'm totally frustrated for this lack of action, that Alan is still in the same situation that he was four years ago."

Cuban authorities arrested Gross in 2009, holding him for more than a year before formally accusing him of illegally distributing prohibited satellite communications equipment to Cuban dissidents.

After a brief trial in 2011, he was sentenced to 15 years in prison.

According to USAID, Gross was in the country to work on a U.S. government-funded project to set up individual satellite-based Internet connections in Cuba. His family has said he was working to help Jewish groups set up Web access.

Despite saying that it plans to provide Cubans with Internet access, the country's government restricts the ability of everyday citizens to access the Web.

The White House did not immediately comment on Gross' letter, but Secretary of State John Kerry said Tuesday in Brussels that the administration is "currently engaged in some discussions" regarding Gross' imprisonment. He said he was not at liberty to go into any detail.

"We will do everything we can and will continue to, but these things are often best resolved in quiet diplomacy, under the radar screen, behind the scenes, and that is exactly what we have been pursuing."

Numerous U.S. representatives have tried to bargain for the former contractor's release, including former New Mexico Gov. Bill Richardson and former President Jimmy Carter, but to no avail.

On Monday, the State Department released a statement calling Gross' continued imprisonment "gravely disappointing."

"Mr. Gross is a 64-year-old husband, father, and dedicated professional with a long history of providing aid to underserved communities in more than 50 countries," the State Department said. "We reiterate our call on the Cuban government, echoing foreign leaders and even Cuba's allies, to release Alan Gross immediately and unconditionally."

Judy Gross said Tuesday that assurances from Kerry that discussions are ongoing aren't enough.

"This is something that I've heard for four years. It hasn't changed. I don't know what that means, 'doing something.' We have not been told what they're doing," she said. "So at this point we feel that we have to step it up a notch, and we are asking President Obama to get personally involved in the situation and do what needs to be done to get him out."

Obama, she said, has the power to jump start the process and give Cuban officials what they've repeatedly told her they want: a chance to talk with an envoy from the U.S. government.

"I want Obama to take Alan seriously, to take the situation seriously," she said. "He's the leader of the nation. He's the one that could go to the State Department, go to the Justice Department and say, 'Let's make this work.' He could do this tomorrow if he wanted to.

"How can you improve any situation without sitting down and talking? It just doesn't work that way," she said.

At a rally across from the White House on Tuesday, she said U.S. calls for her husband's "unconditional" surrender would amount to a "death sentence" for her husband, intimating that the United States will need to bargain with Cuba for her husband's release.

"He tries so hard to hold on to hope, but it is fading fast," she told protesters.

Gross' family says that he has lost nearly 100 pounds, has developed degenerative arthritis in his leg and has battled depression since his arrest. In a statement read to demonstrators on Tuesday, Gross said his treatment at the prison is "stressfully demeaning and my environment is oppressive."

He does not speak Spanish and so cannot communicate with his guards or the other two inmates jailed with him, Gross said in the statement.

Gross says he is allowed out of his cell for one hour a day, which he spends in a small courtyard with high walls and limited views of the sky and nothing else.

"I don't sleep much, between my arthritis and the lights in my cell, which are kept on 24 hours a day. With the exception of a few phone calls and visits, I am completely isolated from the outside world," he wrote.

"I have lost almost everything in the last four years, most of all time with my family -- my wife, Judy, and my daughters, Shira and Nina. I have had to ask my daughters not to visit because I cannot bear them seeing me like this, a shadow of my former self, surrounded by men with machine guns," he said in his letter to Obama.

"My heart is broken from acute homesickness, loneliness and disappointment from my government," Gross said in his statement to protesters.

Gross, a longtime international development worker, said he was pleading for his release not only for his own benefit and that of his family but also for all Americans working, living or traveling abroad.

"There are countless Americans all over the world, some serving in uniform, others serving in diplomatic or civilian capacities, still others private citizens studying or traveling abroad, and they must not harbor any doubt that if they are taken captive in a foreign land, our government will move heaven and earth to secure their freedom," Gross wrote.