Trump had been scheduled to tour the factory Thursday where he also planned to sign executive orders related to American manufacturing.
The visit had not been publicly announced, but White House staffers were already on the ground in Milwaukee setting up for Trump's planned visit to the factory on Thursday.
White House spokeswoman Stephanie Grisham confirmed Trump is not expected to go to Milwaukee on Thursday.
Harley-Davidson issued a statement Tuesday night saying they "don't have, nor did we have, a scheduled visit from the President this week at any of our facilities."
"We are proud to have hosted Presidential visits at our facilities. Three of the last five presidents -- Ronald Reagan, George W. Bush and Bill Clinton -- have visited us at our facilities. These visits are a testament to the pride and passion of our employees and their great work building Harley-Davidson motorcycles," said Maripat Blankenheim, director of Harley-Davidson's corporate communication. "We look forward to hosting the president in the future."
Large protests have been in the works in recent days, particularly in light of Trump's executive order issuing an immigration ban.
It was the threat of protests, and not Trump's planned signing of executive orders, that made Harley-Davidson uncomfortable, the official said.
The canceled trip highlighted the difficulties Trump will continue to face as he looks to implement his agenda while his presidency is engulfed in controversy.
Trump's decision to temporarily ban citizens of seven Muslim-majority countries from traveling to the US and suspend the admission of all refugees has sparked a wave of protests nationwide -- protests which now appear to have torpedoed the rollout of an unrelated part of his policy agenda.
The travel ban was also met with widespread condemnation by Democrats and even some Republicans who called the move counterproductive to US counterterrorism efforts.
That fiery opposition has already ramped up Democrats' efforts to stall the confirmation of Trump's Cabinet nominees, and could also trickle down to legislative efforts.
CNN's Jim Acosta contributed to this report.