The meeting in a small conference room on the sidelines of the Summit of the Americas came as the two countries work to end the Cold War enmity that had led to a total freeze of diplomatic ties. And while both leaders proclaimed progress had been made, a key stumbling block -- Cuba's place on the U.S. list of countries that sponsor terror -- remained unresolved.
"This is obviously an historic meeting," Obama said at the beginning of his session with Castro, claiming that decades of strain had done little to benefit either Cubans or citizens of the United States.
"It was time for us to try something new," he said. "We are now in a position to move on a path toward the future."
Castro, who earlier in the day said he trusted Obama, acknowledged there would be difficult stumbling blocks as his nation works to repair ties with the United States. But he said those differences could be surmounted.
Raul Castro: 'We need to be patient'
"We are willing to discuss everything, but we need to be patient, very patient," Castro said. "We might disagree on something today on which we could agree tomorrow."
Speaking to reporters after his session with Castro, Obama said the meeting was "candid and fruitful" and could prove to be a "turning point" in his push to defrost ties with Cuba.
But he said he hadn't yet decided whether to remove Cuba's designation as a state sponsor of terror, an outcome that had previously been expected during the summit. The State Department provided Obama with a review of the terror status this week.
"I want to make sure I have a chance to read it, study it before we announce publicly what the policy outcome is going to be," Obama said. "But in terms of the overall direction of Cuba policy, I think there is a strong majority both in the United States and in Cuba that says our ability to engage, to open up commerce and travel and people to people exchanges is ultimately going to be good for Cuban people."
On Friday night, Obama and Castro greeted each other courteously amid an explosion of camera flashes, shaking hands
before dining at the inaugural session of the conference. The two sat at the same table but not directly next to one another.
Before Obama arrived in Panama on Wednesday, he spoke with Castro by phone
, laying the groundwork for what will become a new era of relations between the neighboring countries.
No interest in old battles
"The Cold War has been over for a long time," Obama said during opening remarks at the summit Saturday. "I'm not interested in having battles, frankly, that began before I was born."
That exhortation, however, seemed to be lost on Castro himself, who expanded what was meant to be a six-minute speech into a 50-minute address lecturing leaders on Cuba's revolution and giving a litany of perceived grievances to Cuba over the past 50 years.
But he distinguished Obama from past American presidents, saying he respected Obama's move toward reconciliation.
"In my opinion, President Obama in an honest man," Castro said through an interpreter. "I admire him, and I think his behavior has a lot to do with his humble background."
A U.S. administration official said Castro's long list of grievances was expected, despite the move toward diplomatic ties.
"(What's) unique and new is what he said about the president," the official said of Castro's praise for Obama.
Ending a half century of strife
Obama announced in December that he was seeking to renew diplomatic relations with Cuba after half a century of strife, including eventually opening embassies in Washington and Havana.
His meeting with Castro on Saturday isn't being billed as a formal bilateral session, but Obama's aides are still characterizing the event as the highest-level engagement with the Cuban government since then-Vice President Richard Nixon met with Fidel Castro in 1959.
"We're in new territory here," Ben Rhodes, Obama's deputy national security adviser, said Friday. "The reason we're here is because the President strongly believes that an approach that was focused entirely on isolation, focused entirely on seeking to cut off the Cuban people from the United States of America had failed."
Some skeptics in the United States
The overtures to Cuba have not been universally popular in the United States; some lawmakers were irate that Obama was seeking to engage what they regard as a corrupt government.
"A recommendation to remove Cuba from the list of State Sponsors of Terrorism would represent another significant misstep in a misguided policy," Sen. Bob Menendez, a Democrat who used to the chair the Foreign Relations Committee, wrote in a statement last week.
In Latin America, however, Obama was receiving a warm welcome after announcing he was seeking to engage Havana in talks over reopening embassies and removing barriers to commerce and travel.
He noted to applause during a session Friday that this was the first summit with Cuba in attendance. And he's cast the decision to reopen the U.S. relationship with Cuba as beneficial to the entire hemisphere, which has also embraced his immigration executive action.
But even as Obama landed in Panama, the longstanding gulfs between the two countries' governments were on display. Dissidents opposed to Castro's regime were violently accosted this week by supporters of the Cuban government, a scuffle the White House said was unacceptable.
"As we move toward the process of normalization, we'll have our differences, government to government, with Cuba on many issues -- just as we differ at times with other nations within the Americas, just as we differ with our closest allies," Obama said at a meeting of civil society leaders Friday. "There's nothing wrong with that."
"But I'm here to say that when we do speak out, we're going to do so because the United States of America does believe, and will always stand for, a certain set of universal values," he said.
News conference brings out other topics
Obama closed out his time in Panama with a news conference where he covered topics ranging from Hillary Clinton's expected presidential announcement to his framework deal with Iran on its nuclear program.
The President had pointed criticism for Sen. John McCain, R-Arizona. Earlier this week, McCain accused Secretary of State John Kerry of intentionally mischaracterizing what the sides had agreed to in the Iran nuclear deal.
"John Kerry is delusional," McCain said on the Hugh Hewitt show, a conservative talk radio program, adding that the view from the Supreme Leader of Iran of the provisions agreed to "is probably right," rather than what the United States maintains are the agreed provisions.
While discussing the Iran agreement Saturday, Obama brought up those remarks without being asked.
"When I hear someone like Sen. McCain recently suggest that our secretary of state, John Kerry, who served in the United States Senate, (is) a Vietnam veteran, who's provided exemplary service to this nation, is somehow less trustworthy of the interpretation of what's in a political agreement than the Supreme Leader of Iran, that's an indication of the degree to which partisanship has crossed all boundaries," he said at the news conference.
After the President's remarks, McCain tweeted "So Pres. Obama goes to #Panama, meets with Castro and attacks me - I'm sure Raul is pleased."
As for his 2008 Democratic rival, Obama said, "If she decides to run, if she makes an announcement, she's going to have some strong messages to deliver," he said.