Merkel, who after 12 years in power held a double-digit lead for most of the campaign, scored around 33 percent of the vote with her conservative Christian Union (CDU/CSU) bloc, according to exit polls.
Its nearest rivals, the Social Democrats and their candidate Martin Schulz, came in a distant second, with about 20-21 percent.
But in a bombshell for the German establishment, the anti-Islam, anti-immigration Alternative for Germany (AfD) captured around 13 percent, making it the country’s third biggest political force.
The four-year-old party with links to the far-right French National Front and Britain’s UKIP has been shunned by Germany’s mainstream.
It is now headed for the opposition benches of the Bundestag lower house, dramatically boosting its visibility and state financing.
Alarmed by the prospect of what Foreign Minister Sigmar Gabriel branded “real Nazis” entering the Bundestag for the first time since World War II, the candidates had used their final days of campaigning to implore voters to reject the populists.
“This Alternative for Germany is no alternative. They are a shame for our nation,” former European Parliament chief Schulz told a rally in Berlin on Friday.
Germans elected a highly splintered parliament reflecting an electorate torn between a high degree of satisfaction with Merkel and a desire for change after more than a decade of her leadership.
Another three parties cleared the five-percent hurdle to representations: the liberal Free Democrats (around 10 percent), anti-capitalist Left and ecologist Greens (both at about nine percent).
As Merkel failed to secure a ruling majority on her own, the process of coalition building was shaping up to be a thorny, potentially months-long process.
Merkel, 63, whose campaign events were regularly disrupted by jeering AfD supporters, said in her final stump speech in the southern city of Munich that “the future of Germany will definitely not be built with whistles and hollers”.
Merkel, often called the most powerful woman on the global stage, ran on her record as a steady pair of hands in a turbulent world, warning voters not to indulge in “experiments”.
Pundits said Merkel’s reassuring message of stability and prosperity resonated in greying Germany, where more than half of the 61 million voters are aged 52 or older.
Her popularity had largely recovered from the influx since 2015 of more than one million mostly Muslim migrants and refugees, half of them from war-torn Syria, Iraq and Afghanistan.
But the AfD was able to capitalise on a wellspring of anger over the asylum issue during what was criticised as a largely lacklustre campaign bereft of real clashes among the main contenders.
While the likelihood of the AfD winning seats was clear for months, commentators called its strong showing a “watershed moment” in German post-war politics.
The party has made breaking taboos its trademark.
One of two AfD leading candidates, Alexander Gauland, has called for Germans to shed their guilt over two world wars and the Holocaust and to take pride in their veterans.
He has also suggested that Germany’s integration commissioner Aydan Ozoguz, who has Turkish roots, should be “disposed of in Anatolia”.
Law student Sabine Maier dismissed the AfD as “too extreme” as she voted in Berlin.
But she also criticised the media for lavishly covering the most outrageous comments by the upstart party.
“They aren’t all fascists,” she said.
Meanwhile Schulz, 61, struggled to gain traction with his calls for a more socially just Germany at a time when the economy is humming and employment is at a record low.
The SPD also found it hard to shine after four years as the junior partner in Merkel’s left-right “grand coalition”, marked by broad agreement on major issues, from foreign policy to migration.
In the final stretch, the more outspoken Schulz told voters to reject Merkel’s “sleeping-pill politics” and vote against “another four years of stagnation and lethargy”.
The CDU and the SPD have signalled they aren’t keen to continue their loveless marriage.
The SPD’s catastrophic result may convince many rank-and-file members that the traditional working-class party would benefit from a stint in opposition to rekindle its fighting spirit.
This would leave Merkel in need of new coalition partners — possibly the pro-business Free Democrats, who staged a comeback after crashing out of parliament four years ago.
In theory they could join forces with the left-leaning Greens, who, however, starkly differ from the FDP on issues from climate change to migration policy.