"It makes it harder to unite," said Rep. Raul Labrador, an Idaho Republican and leader within the conservative House Freedom Caucus.
Nevada Rep. Mark Amodei was incredulous about the speaker's high profile break with Trump last week, saying, "Ryan goes out on CNN and says it's Donald Trump's duty to unite the party," asking, "So Mr. Speaker, what's your job regarding unity?"
After the meeting, Ryan waved off concerns he was making things tougher, saying, "To pretend we are unified without actually unifying means we go into the fall at half-strength. This election is too important to go into an election at half-strength."
Many House GOP members shared some of Ryan's concerns about the controversial billionaire, but after his lengthy string of convincing primary victories, they are resigned to lining up behind Trump.
But among conservatives there are still reservations, especially about the evolving positions the GOP nominee has taken on a broad range of issues. At Wednesday's closed-door meeting, several members urged Ryan to raise policy priorities they have championed when he sits down with Trump on Thursday.
Specifically, they want Ryan to press Trump about his positions on abortion, budget and spending issues, and reforms to Medicare and Social Security. Other Republicans urged Ryan to caution the presumptive nominee that if he makes more controversial statements, like suggesting riots would occur if he were denied the GOP nomination, it will make those running on the same ticket as Trump in November uncomfortable to link themselves to their nominee.
North Carolina GOP Rep. Walter Jones said after the meeting, "Mr. Trump has different positions and they change every so often, so we try to figure out where we're going." He defended Ryan taking some time to work through his position, saying, "I don't think there's any other choice than what he's doing."
Despite lawmakers' deep concerns about Trump, many said they were backing him because Democrat Hillary Clinton would be worse in the White House.
"Even if Donald Trump brings chaos to this party, that's better than Hillary Clinton bringing destruction to this Constitution," Arizona GOP Rep. Trent Franks said.
Some were willing to let the process with Ryan and Trump play out.
"This looks to me like one of those political dances that ends up OK in the end. But you do have to go through the dance, and there's always a chance for a misstep," said Rep. Tom Cole, an Oklahoma Republican and close ally of Ryan.
The next step in that dance, he said, would be Trump meeting with the entire House Republican Conference, something lawmakers pitched to Ryan during their Wednesday meeting.
'Men are meeting'
Ben Carson, acting as an emissary for Trump, spoke to Ryan on Tuesday. Armstrong Williams, Carson's business manager, told CNN's Wolf Blitzer on Wednesday the call was meant to assure Ryan that "Donald Trump is a reasonable man."
Williams said he didn't expect the Thursday meeting to yield an endorsement, but did suggest that Trump and his associates were sympathetic to Ryan's concerns.
"You can't take offense at the speaker being candid, just like Mr. Trump speaks candid," Williams said on "Wolf." "You've gotta respect that. So what the speaker was saying is, 'Listen, we've got a down-ticket. There are people who are nervous about the kinds of things Mr. Trump has said in the past.'"
Williams added: "Men are meeting. Both with very critical roles in this country. They should sit down and talk like men. Unify their party. Show respect."
Many lawmakers have also said they want to hear more from Trump about how he would promote conservative principles and where specifically he stands on issues.
Members of the House Freedom Caucus, who many credit with pressuring former House Speaker John Boehner into retirement, said they want assurances from Trump he will support conservative ideals. Top leaders in the House Freedom Caucus have requested their own meeting to discuss these issues with Trump directly.
Rep. David Brat, a Virginia Republican and Freedom Caucus member, said that Ryan and Trump should sit down Thursday and work up a new "Contract with America" -- a nod to former House Speaker Newt Gingrich's 1994 plan -- and present it as a unified plan from the Republican Party.
Louisiana GOP Rep. John Fleming, who told reporters he's backing Trump, said the process for his colleagues could take months and stretch out through July when Trump will get the formal Republican nomination. But he warned, "I really think everything has to be resolved by the end of the convention."
Rep. Chris Collins, the first House Republican to endorse Trump, downplayed others' criticisms of the speaker, saying he and others who are still not backing the nominee thought the race would last longer and need time, but he told reporters, "we'll be there."
Fleming also said he believes Ryan and Trump will work out their differences, saying about the billionaire turned politician, "he's a dealmaker -- he wants to find a resolution."