(CNN) -- Embattled New York Gov. David Paterson vowed Sunday he won't give in to White House pressure to drop out of next year's election, insisting, "I am not failing to stand up for my party."
New York Gov. David Paterson, right, with President Obama last week in Albany, New York.
The nation's first legally blind governor is facing approval ratings dipping below 20 percent.
"I'm blind but I'm not oblivious," he said Sunday on NBC's "Meet the Press." "I realize that there are people who don't want me to run."
Last Sunday, The New York Times reported that the White House was urging Paterson to withdraw. The White House neither confirmed nor denied that to CNN.
But informed sources told CNN that White House political director Patrick Gaspard met with Paterson to let him know about the administration's concerns he could not win the governor's race next year. That loss could affect races down-ticket, including the Senate seat now held by Kirsten Gillibrand.
"I have spent a whole life being told I couldn't do things. I was told by guidance counselors I shouldn't go to college," Paterson said on Sunday. "I was told when I was minority leader of the Senate that we couldn't win the majority. We won eight seats in four years and won the majority."
The governor also noted that a court this week upheld his appointment of a lieutenant governor, disproving those who said it would not be upheld. The message to him, Paterson said, "was that you don't give up."
In the NBC interview, Paterson left his descriptions vague about discussions with the White House, saying he would not detail "confidential conversations."
He said "the president has never told me not to run for governor" and that he's "never gotten an explicit indication authorized from the White House that I shouldn't run."
"They certainly sent a message that they have concerns. And I appreciate that," he said.
He acknowledged that it has been "a difficult week."
Asked about his wife's remarks to the New York Post that he had been "stunned" by the message from the White House, Paterson responded, "Michelle is very protective of me. I don't know that I was stunned."
"I am not failing to stand up for my party -- I fight for the priorities of my party," he said, adding that he is "fighting for the people of the state of New York. ... So I'm not going to run away from a fight when I know who I'm fighting for."
Throughout the interview, Paterson cited the economic crisis and his need to cut $30 billion in spending. He said the cuts have brought on "a hail of criticism" from special interests, "and that is what accounts for the low poll numbers."
One of only two African-American governors, Paterson said he does not believe race has been a factor in the poll numbers.
Paterson was lieutenant governor when a scandal led to then-Gov. Eliot Spitzer's departure from the office in March 2008.