(CNN) -- President Obama is vowing the loss of a Republican in his Cabinet will not stop his efforts to bridge the partisan divide.
Sen. Judd Gregg says it was his mistake to accept the offer from President Obama.
Sen. Judd Gregg said Thursday he was withdrawing his nomination as commerce secretary, citing "irresolvable conflicts" over the administration's stimulus bill and the upcoming 2010 census.
Gregg didn't like some of the spending in the stimulus plan and had concerns the White House would take control of the census away from him, sources said.
The population count determines how congressional districts are drawn, sparking Republican fears that the Obama White House was going to tilt the balance of power.
Gregg, R-New Hampshire, said he realized that his withdrawal is "unfair in many ways," but he said it would have been an even bigger mistake to take the job "and then find myself sitting there and not being able to do the job the way it should be done on behalf of the president."
Gregg said it was "my mistake, obviously, to say yes."
A Republican source said Gregg "didn't want to be a powerless GOP token and that's where this was headed."
A source close to the process said the idea of removing the census from the Commerce Department and having it "run" out of the White House was taken as a slap in the face by Gregg.
The Obama administration has said it has no intention of doing so. While it isn't unusual for White House staff to have input into the census process, Obama has taken no official action to remove control of the process from Commerce.
The source said those who wanted Gregg -- including Chief of Staff Rahm Emanuel -- believed that in choosing him, the Obama team would be "taking the opposition's quarterback off the field."
"They have to take some of the blame for this," a Democratic source close to the White House said. "They almost humiliated him by taking the census away from him."
However, the source added, Gregg's decision to recuse himself on the stimulus vote was not taken well at the White House either -- it was seen as an insult.
Gregg did not cast a vote when the Senate voted on its version on the stimulus package earlier this week.
Obama admitted to the (Springfield, Illinois) State Journal-Register that the move was "something of a surprise," but the self-proclaimed "eternal optimist" tried to shrug it off.
At the Abraham Lincoln Association dinner, Obama joked, "In 1854, Lincoln was simply a Springfield lawyer, who'd served just a single term in Congress.
"Possibly in his law office, his feet on a cluttered desk, his sons playing around him, his clothes a bit too small to fit his uncommon frame, maybe wondering if someone would call him up and ask him to be commerce secretary," Obama said Thursday night.
Obama earlier praised Gregg as "good man" who "sincerely wanted to work with us."
"I think he had a change of heart about the idea of leaving the Senate. The one thing I want to make sure of these people don't take from this the notion that we can't get Democrats and Republicans working together," he said onboard Air Force One.
David Gergen, a senior political analyst for CNN, said while the shakeup is embarrassing for Obama, it could be more damaging for Gregg since he reportedly lobbied for the position. Watch more on the fallout from the withdrawal »
White House press secretary Robert Gibbs said Thursday that Gregg initiated discussions over the position and made it clear that "despite past disagreements about policies, he would support, embrace, and move forward with the president's agenda."
Gergen said, "I think it was awkward for both sides ... and I think that raises questions about whether the Obama team did their homework."
He added, "But I think it's more of an embarrassment for Judd Gregg. I do not think it's a war against the White House. I think it underscores the political culture in Washington and the philosophical divides -- [there are] deep, deep divisions, which the election of Barack Obama has not yet healed."
Gregg is Obama's third Cabinet nominee to withdraw, following last week's decision by Health and Human Services Secretary-designate Tom Daschle to quit over tax issues and the withdrawal of Obama's previous Commerce Department pick, New Mexico Gov. Bill Richardson.
Gregg would have been the third Republican to join the Democratic administration, following Defense Secretary Robert Gates and Transportation Secretary Ray LaHood.
With Gregg out, the Obama team loses someone who could have leaned on Senate conservatives with concerns about spending more money on the second round of bank bailouts and the foreclosure crisis, coming on the heels of the $789 billion economic stimulus plan.
That framework for bipartisanship is gone now, and there's another vacant Cabinet spot.
John King, CNN's chief national correspondent, said the blame falls on both sides.
"So there were red flags in this relationship from the beginning, and it appears that both sides did not deeply explore them enough. And if Sen. Gregg decided in the end he was not comfortable with it, that is his right.
"But the question being asked in Washington, especially after the Richardson nomination, after the Daschle nomination and after the controversies about [Treasury Secretary Timothy] Geithner and other Obama picks, why didn't somebody at the White House push, push, push and push on all these issues, just to make sure?"
Both sides were taking heat for the arrangement: Gregg from Republicans questioning why he would join the rival team and Obama from people questioning why someone who voted to abolish the census would be tapped for the Commerce Department. Watch more on the pressure from the GOP »
Gregg, a member of the Senate Appropriations Committee and the subcommittee that oversees the Commerce Department, cast a vote in 1995 to abolish the entire department and in 1999 opposed emergency funding for the 2000 census.
Gergen, who served in the Nixon, Ford, Reagan and Clinton administrations, said while each party has "hotheads" who would like to destroy the opposition, the source of the problem boils down to the political culture.
Gregg, he said, would not have been able to go in for the best interests of the country and work in a Democratic administration without being treated like a traitor.
"It's not the people trying to pull him back in. It's the ostracization. I know something about this. I've gone through this. And you feel like you're suddenly isolated, that you're in no man's land because you've agreed to work with the other side. That speaks to the health of the political culture," he said.
But Gergen said there's a silver lining. In his opinion, with two commerce secretary picks out, the Obama team sees what does not work: a political figure at the Commerce Department.
"What they need there is what the Commerce Department has traditionally had. They need a CEO in this administration," he said. "And I think they've got the opportunity now to put the kind of person they should have put in there to begin with."
CNN's Gloria Borger, Ed Henry, Kristi Keck and Jessica Yellin contributed to this report.