Preliminary results showed Merkel's CDU receiving 33% of the vote in North Rhine-Westphalia, or NRW, followed by the Social Democratic Party's 31.4%.
It's a major upset for the SDP, in a state it has governed almost without interruption since 1966. NRW is Germany's most populous state and the home of SPD national leader Martin Schulz, the former EU Parliament president who will face Merkel in federal elections later this year.
CDU General Secretary Peter Tauber crowed over the election results at a party gathering in Berlin, saying his party had "conquered the heart" of the SPD.
"It's a difficult day for the SPD, a difficult day for me personally," Schulz told supporters in Berlin.
Schulz said he has learned from Sunday's loss and signaled that's he's ready to answer critics who say he's too agreeable and lacks substance.
"I've heard the criticism of people who say 'you're nice, but you have to get more specific.' And that's what I plan to do," Schulz told CNN affiliate ARD.
Schulz defiantly asserted that the SPD is a party that knows how to fight back and noted, "The federal elections are still a long way off."
More momentum for Merkel
The NRW state election is the last major political contest in Germany before the national vote in September. Sunday's results mark the third victory in recent state elections for Merkel's party, giving the German Chancellor another shot of momentum heading into the nationwide vote.
Merkel's CDU saw solid gains when it won elections in Germany's small southwestern state of Saarland
in March and also beat out the SPD in the northern state of Schleswig-Holstein on May 7.
Sunday's win in NRW comes a day before Merkel is scheduled to meet in Berlin with France's newly installed President Emmanuel Macron
Another toehold for Germany's far-right
Germany's right-wing nationalist party, Alternative für Deutschland, won a little over 7% of Sunday's vote and will therefore have representation in the NRW parliament.
The AfD is now represented in 13 states across Germany. The party landed an unprecedented victory last year by defeating Merkel's party in her home state of Mecklenburg-Vorpommern.
That election last September was viewed by many as a referendum on Merkel's immigration policy and a demonstration of the gaining steam of anti-immigrant parties in Europe.
The far-right wave has waned in recent European elections, however.
In the Netherlands in March, populist candidate Geert Wilders was soundly beaten by Conservative Prime Minister Mark Rutte. Last week, new French President Macron won a landslide victory
over far-right opponent Marine Le Pen.