The at-large congressman from North Dakota also reiterated that he hasn't ruled out a Senate bid next year for the seat currently held by Democratic Sen. Heidi Heitkamp, and he said Trump has already pledged his support should Cramer decide to run.
The women, who represented the House Democratic Women's Working Group, said they were wearing white not only in memory of the suffrage movement but also to show Trump their support for a number of issues affecting women, such as affordable health care, reproductive rights, equal pay and paid leave. The effort was also a nod to the start of Women's History Month, they said.
"That was obvious, not just, not by their clothes, but in addition to their clothing, their gestures, their hand gestures, their thumbs down, their quick exit from the gallery ahead of the President," he said. "Their behavior in general."
Earlier Wednesday, Cramer poked fun at the women's outfits -- which consisted of a mix of white dresses, dress suits and pantsuits -- during a radio town hall, saying their coordinated effort was akin to a "disease."
"But by the way, did you notice how poorly several of them were dressed as well?" he said, responding to a constituent on the call. "It is a syndrome. There is no question, there is a disease associated with the notion that a bunch of women would wear bad-looking white pantsuits in solidarity with Hillary Clinton to celebrate her loss. You cannot get that weird."
Cramer acknowledged some Democrats stood up and applauded during parts of Trump's speech, and said both parties could work together on issues like jobs and infrastructure.
Rep. Debbie Wasserman Schultz responded to Cramer's comments during CNN's "Erin Burnett OutFront" on Wednesday, saying the North Dakota Republican just didn't get it.
"He obviously misses the point," the Florida Democrat told Burnett. "When we are sitting right in front of (Trump) with a sea of white attire, that we are not going to allow him to roll back women's progress in this country, it's actually patriotic and shows that we care about the issues that are important to women and won't let them roll back our progress."
Wasserman Schultz jokingly added that she chose to wear a sleeveless white dress -- "that I have gotten lots of compliments on."
House Minority Leader Nancy Pelosi responded to Cramer's comments on Twitter.
"Thank you for illustrating why we so badly need to honor #WomensHistoryMonth," Pelosi tweeted
, along with a link to a Politico article about his comment.
Rep. Lois Frankel, D-Florida, leads the working group and said in a statement that "instead of being a fashion critic, Mr. Cramer should pay attention to our message."
She added that they wore white "in solidarity with the suffragists in unity against Republican attempts to roll back the hard earned progress that has been made on behalf of women and girls."
Meanwhile, Rep. Cheri Bustos, D-Illinois, tweeted
a sarcastic comment about Cramer. "The @HouseGOP is off to a great start for #WomensHistoryMonth."
Cramer, speaking to CNN, said his comment in the radio town hall was "really more metaphorical than literal."
"But at the same time, they looked silly," he said, again stating that the women were being rude.
"I don't buy their argument that it was a celebration of suffrage. I think they should be celebrating the fact that there were women members of Congress sitting in a joint session, listening to the President of the United States on equal footing as a co-equal branch -- and sort of get over this notion that somehow we have to be offended all the time."
As for a 2018 Senate bid, Cramer said he has not "ruled it out," but added that it's "not a top-of-the-mind issue for me right now, and it won't be for several months."
He added that Trump has spoken to him about the topic. "He's pledged his support should I run for the Senate -- and in a big way."
CNN has reached out to the White House to confirm that Trump has already committed his support.