Six of his teammates -- Devin McCourty, Martellus Bennett, Chris Long, LeGarrette Blount, Alan Branch, Dont'a Hightower -- have said they plan on skipping the expected upcoming visit.
Bennett, McCourty and Blount directly cited their opposition to Trump as a reason for their boycott, but Brady said the tradition of Super Bowl champs visiting the White House each year "never really was a political thing" for him.
"Putting politics aside it never really was a political thing -- at least it never was for me," Brady told NBC Sports
. "It was just always something that was a privilege to be able to do because it really meant you won a championship and you got to experience something cool with your team and your teammates."
The five-time Super Bowl champ has been friends with President Donald Trump for years. He noted his own record of missing a White House visit -- in April 2015, when the team was honored by President Barack Obama.
"Everybody has their own choice. There's certain years, like a couple years ago, I wanted to go, and then didn't get the opportunity based on the scheduling," Brady said. "We didn't get told until 10 days before we were going and at that point I had something that I had been planning for months."
At the time, a Patriots spokesman told ESPN
that Brady had a "family commitment.
The Patriots have been linked to Trump because of the President's friendship with Brady, head coach, Bill Belichick, and owner Robert Kraft. Kraft joined Trump last week at Mar-a-Lago, where they dined with Japanese Prime Minister Shinzo Abe.
Kraft downplayed reports
of his players skipping the White House visit during an interview on NBC's "Today" show Monday, saying that various players choose to skip the White House visit each time but "this is the first time it's gotten any media attention."
"Every time we've had the privilege of going to the White House, a dozen of our players don't go,'' Kraft said. "This is the first time it's gotten any media attention ... Some of the players have the privilege of going in college because they're on national championship teams, (and) others have family commitments.''
The New York Times last month that Trump is going "to be great for the economic side of America."
"Loyalty is important to me, and he has been a wonderful friend," Kraft said. "I think one of the great problems in the country today is the working poor, the middle class, that there hasn't been growth in income on an equal basis, and I really think the policies he's going to bring to bear are going to be great for the economic side of America."
But Brady, who faced controversy during the 2016 election when a "Make America Great Again" hat was spotted
in his locker, has been evasive when asked about Trump, keeping his comments non-political.