(CNN) -- Detained American Alan Gross told his wife and daughter something they weren't expecting during a recent visit in Cuba: Goodbye.
"Our daughter, Nina, was unprepared to see how gaunt and physically frail her father has become. And his decision to say goodbye to us was wrenching," Judy Gross said in a statement released Monday.
The former U.S. Agency for International Development subcontractor, who's been imprisoned for nearly five years, is refusing to see visitors from the U.S. Interests Section in Havana, the statement said.
"Alan has withdrawn, and he told me that his life in prison is not a life worth living," said Scott Gilbert, Gross's attorney.
Gross, 65, is serving a 15-year sentence for bringing satellite communications equipment to Cuba as part of his work as a subcontractor for the U.S. Agency for International Development. He was convicted in March 2011.
U.S. officials said Gross was merely trying to help Cubans bypass the island's stringent restrictions on Internet access. But Cuban authorities say Gross was part of a plot to create "a Cuban spring" and destabilize the island's single-party communist government.
A letter to U.S. President Barack Obama from 300 rabbis across the United States Monday called for the U.S. government to negotiate his release, describing the situation as "increasingly urgent."
"Alan went to Cuba on behalf of our government. His immediate release from prison in Cuba and return to the U.S. must be a priority for our nation," the letter says. "Indeed, we believe this is a moral imperative."
Asked Monday about reports that Gross had refused to meet with the new head of the U.S. mission in Havana, State Department spokeswoman Jen Psaki said she couldn't provide details due to privacy restrictions.
"Absent written authorization, there's really not more information I can share about those specific reports," she said.
U.S. officials have repeatedly pushed for Gross's freedom, she told reporters.
"We keep his case at the forefront of discussions with the Cuban government, make clear the importance the United States places on his welfare, and we engage also with a range of our foreign counterparts at the highest levels and urge them to advocate for his release," she said. "So we urgently reiterate our call for the Cuban government to release him immediately."
Last month Judy Gross told CNN that her husband had reached his breaking point.
In April, Gross went on a hunger strike to protest the lack of progress between the United States and Cuba over his case.
Gross finally heeded his mother Evelyn's request to end the protest.
Last month, Evelyn Gross died from cancer. Cuban officials denied his request to attend her funeral.
"Alan's emotional deterioration has been severe," Gilbert said Monday, "and his mother's lingering and painful death has only accelerated this."
After Bergdahl, will U.S. and Cuba make swap involving Alan Gross?
CNN's Patrick Oppmann, Kevin Liptak and Elise Labott contributed to this report.