"I haven't thought about it. I am not going to go," the New England Patriots tight end said. "I can elaborate later on in life; right now I am just trying to enjoy this ... People know how I feel about it, just follow me on Twitter."
Bennett added that he is "not really worried" about potential backlash over skipping the White House visit from the team's owner, Robert Kraft, and said that he and his teammates avoid talking politics in the locker room.
"You just don't bring that to work. We all have our beliefs. We accept people for who they are," he said.
CNN has reached out to the White House for comment and have not yet received a response.
Bennett is an outspoken supporter
of the "Black Lives Matter" movement and has slammed Kanye West for meeting with Trump
"So Kanye didn't take the time to vote. And now he holding on to Trump's coattail like Peter Pettigrew to Lord Voldemort. Kanye Pettigrew," Bennett tweeted.
After San Francisco 49ers quarterback Colin Kaepernick sat during the national anthem
to protest racism in America in September, Bennett participated in a similar protest on the field, raising his fist with fellow teammate Devin McCourty during the national anthem at a September game.
Fist-raising is a symbol of solidarity and a symbol of the black power movement that dates back to the Black Panthers in the 60s and 70s.
The Patriots have been linked to Trump due to the President's friendship with quarterback Tom Brady, who faced controversy during the 2016 election when a "Make America Great Again" hat was spotted
in his locker in September.
Brady has been evasive when asked about Trump and has kept his comments on the issue non-political.
a day before the general election that Brady and the team's coach Bill Belichick are supporting his campaign for president.
But the football star's wife, model Gisele Bundchen, bluntly denied those reports in a social media exchange.
"Gisele I head you and Tom were backing Trump! Is that true?" a Instagram user asked.
"NO!" Bundchen responded.
However, following Trump's win, the President said at a pre-inaugural dinner that Brady and Patriots owner Robert Kraft called to congratulate him.
While he did not deny it, Brady played down
the claim during an interview, saying, "I call a lot of people."
"I have called him, yes, in the past," he said. "Sometimes he calls me. Sometimes I call him. But, again, that's been someone I've known. I always try to keep it in context because for 16 years you know someone before maybe he was in the position that he was in. He's been very supportive of me for a long time. It's just a friendship. I have a lot of friends. I call a lot of people."