(CNN) -- Sixteen-fifty-one Pennsylvania Avenue isn't quite as famous as the address right across the street.
But on Thursday the four-story townhouse will be in the spotlight, as Blair House plays host to the nationally televised summit on health care with President Obama and congressional leaders.
Blair House is officially the president's guest house, operated by the State Department as the home to international dignitaries during visits to Washington. But from efforts to stop the Civil War to decisions on firing a popular general, and even a dramatic assassination attempt, the building has been the scene of historic moments that go far beyond a diplomatic hotel.
The building has stood since 1824, built as a home for the first surgeon general of the U.S. Army. But a decade later, it was bought by the politically powerful Blair family, as Francis Preston Blair became a key backer of President Andrew Jackson.
Blair's rise began as he wrote stories defending the Jackson administration as a newspaper editor in Kentucky. Candace Shireman, the curator of Blair House, said Jackson was "getting pummeled in the papers" in Washington and looking for a stronger voice to run a pro-administration newspaper. Blair was invited to Washington by the president, and became that voice, taking control of the Globe newspaper.
Amid deep divisions in his administration, Jackson turned to his "kitchen cabinet" of unappointed advisers, with Blair a central figure.
"He was extremely powerful," Shireman said, "moving behind the scenes." Blair and his family "had a tremendous impact on American history, but are hardly mentioned in the history books."
Through the years, Blair and his sons became confidants, advisers and friends to several presidents. President Abraham Lincoln named Montgomery Blair to his cabinet and was a frequent visitor to the house.
The gatherings there, Shireman said, were often "business and politics," but also a chance to get away from work or family pressures. Shireman described Blair House as a respite for Lincoln, where "he knew there would a confidential and sympathetic ear."
As the nation neared civil war, Shireman said Lincoln turned to Blair "to try to broker peace with the Confederacy." He asked Blair to meet directly with Jefferson Davis several times. Shireman said Lincoln believed "he could be very persuasive," though the attempts failed.
Blair House was at the center of another critical moment ahead of the war: It was the setting for "a last-ditch effort" to persuade Robert E. Lee to take command of the Union Army and not fight with the South.
Shireman said Lincoln was sensitive that the decision was a difficult one for Lee, then head of the U.S. Cavalry but a native of Virginia. Rather than the request coming from the commander in chief, he again asked the elder Blair to hold a small private meeting with Lee.
"He thought a family connection would help," she said, with Blair's daughter married to Lee's distant cousin. But ultimately, Lee said he could not fight against his family in the South and resigned his commission.
Long a scene of presidential visits and meetings, Blair House didn't become part of the U.S. government until 1942, when President Franklin Roosevelt saved it from the wrecking ball. Shireman said Roosevelt invoked the War Powers Act to help get the townhouse and turn it into the official guest house.
"He knew diplomacy would be key to ending [World War II]," she said. Working with coalition leaders from overseas, Shireman said he needed a place "to house, entertain, extend hospitality" to the leaders. He couldn't keep them all at the White House.
The story goes that Eleanor Roosevelt played a big part in that decision after an early morning encounter with British Prime Minister Winston Churchill, a frequent visitor to the White House to discuss war plans. Former Secretary of State Warren Christopher, welcoming VIPs to Blair House, told them the "Washington lore:"
"At 6 o'clock in the morning, Mrs. Roosevelt found Churchill, dressed in his robe, pacing excitedly outside the master bedroom, wanting to talk with the president. Given FDR's poor health, Mrs. Roosevelt's first thought was to protect her husband's sleep. She was already upset that Churchill had kept FDR up half the night talking about the war. At that moment she decided to have the government buy a separate building for visiting dignitaries."
Blair House, now government property, became the de facto White House for President Truman while the real thing was undergoing major renovations. His family moved into Blair House from 1948 to 1952, and it served as the Truman White House for meetings.
The dining room became the Cabinet meeting room. Shireman said the Marshall Plan for post-war Europe was created in meetings there. It was there that the Truman Doctrine, the centerpiece of Cold War policy, was born after World War II.
Shireman said Blair House was the site of meetings as Truman decided to fire Gen. Douglas McArthur during the Korean War.
Blair House was also where two Puerto Rican nationalists tried to assassinate Truman, attacking the building on November 1, 1950, as Truman was upstairs. The Blair House Web site describes the attack:
"The two men approached the front door of Blair House from opposite directions, with the plan of shooting their way into the house. A gunbattle ensued on and around the steps of Blair House, with White House police officers and Secret Service agents defending the door. President Truman, who was napping in an upstairs bedroom, was awakened by the sound of gunfire."
One account said Truman "ran to the window and was immediately yanked away and whisked to safety." But Secret Service agent Floyd Boring in an oral interview with the Truman Library said the the president did not go to the window to see what was going on.
Boring, the first person to talk to Truman after the gunfire stopped, told the Library that Truman's first words were "What the hell is going on down there?"
White House police officer Leslie Coffelt, on duty that day at Blair House, was wounded in the shootout and died hours later. His badge is kept in the building's security office. Shireman said he was the only White House police officer ever killed protecting the president.
The shootout left one of the attackers dead and the second wounded.
Through the years, presidents have gone to Blair House to greet arriving leaders, including President Clinton meeting South African President Nelson Mandela there in 1994.
Clinton also held Cabinet meetings there, including the first of his second term in office in 1997. Clinton and Vice President Al Gore presented the results of their "Reinventing Government" project there, with the goals published in what they called "The Blair House Papers."
Obama used Blair House to host a closed-door retreat with his Cabinet at the six-month mark of his first year in office. They had dinner there the opening night and the Cabinet took part in a daylong policy session on August 1.
Presidents-elect also use Blair House as temporary living quarters just before they are sworn into office, often making the Inauguration Day morning walk across Pennsylvania Avenue to their new home to meet the outgoing president and first lady.
Obama's stay at Blair House with his family was pushed back slightly, because the house was being used for a few days by Australian Prime Minister John Howard and pre-arranged guests. He had no trouble securing the location as president, though, for the health care summit.
Thursday's meeting will be held in the Garden Room, a relatively new addition to Blair House. It was built onto the site in a renovation project during the Reagan administration in the 1980s.
The gathering is a rare time that television cameras will be inside Blair House. The meeting with cameras rolling comes after Obama was initially criticized for not fulfilling a campaign promise to have health care negotiations broadcast on television, part of the administration's transparency pledge.
Ironically, one of the first efforts to shed light on what happens in Washington started at Blair House in 1837.
Francis Preston Blair and his publishing partner, John Rives, started a second national publication along with their newspaper, called the Congressional Globe. Shireman said it was a record of congressional records and proceedings, which lives on today as the Congressional Record.
"Before TV, radio and newspaper, it was designed to show people what elected officials were doing or not doing for them," Shireman said.