During an interview on CNN's "OutFront with Erin Burnett" on Friday night, Garamendi began by calling the President "crazy" and comparing him to Kim Jong Un, the North Korean dictator.
"We've got a crazy leading the North Korea and we've got a crazy in the White House," Garamendi said. "We got a guy in the White House who's unstable and not fit for office."
Garamendi said leaders of the Republican caucus in Congress have agreed with him in private conversations that Trump's mental fitness is a concern. He went on to say that Republicans aren't saying this publicly, despite agreeing behind closed doors, because they have a legislative agenda to get passed and need the President on board.
"I'm not going to tell you who they are, but I'm telling you, they are key people in the Republican Congress. They're just shaking their heads, saying, 'Oh my God, look what he did today,' " Garamendi said. "They're concerned. They know what's happening."
Asked if the Republicans he spoke of are members of the GOP leadership, Garamendi said: "I think I've gone far enough. Let's just say these are people that are important in the Republican Congress."
The White House did not immediately respond to a request for comment.
Questions about Trump's mental fitness come in the wake of a chaotic picture painted by Michael Wolff's new book, "Fire and Fury: Inside the Trump White House."
CNN has not independently verified all the details in Wolff's book.
On Thursday, White House press secretary Sarah Sanders called it "disgraceful and laughable
" to question Trump's mental fitness to serve.
"If he was unfit, he probably wouldn't be sitting there, wouldn't have defeated the most qualified group of candidates the Republican Party has ever seen," Sanders said, before praising the President as an "incredibly strong" leader.
A dozen lawmakers from the House and Senate received a briefing from Yale psychiatrist Dr. Bandy X. Lee
on Capitol Hill in early December about Trump's fitness to be president -- and Lee has been asked to speak with additional lawmakers, worried about the President's mental state, later this month.
"Lawmakers were saying they have been very concerned about this, the President's dangerousness, the dangers that his mental instability poses on the nation," Lee told CNN in a phone interview Thursday,
"They know the concern is universal among Democrats, but it really depends on Republicans, they said. Some knew of Republicans that were concerned, maybe equally concerned, but whether they would act on those concerns was their worry."
Lee's public comments are unusual, given protocols from medical professional organizations -- including the 37,000-member American Psychiatric Association -- banning psychiatrists from diagnosing patients without a formal examination.