It was the first time a far-right party will enter the German parliament in more than half a century.
Angela Merkel has secured a fourth term as German chancellor but with her authority diminished, after her conservative bloc secured the lead position in parliamentary elections but failed to halt the march of rightwing populists.
Her current coalition partners, the centre-left Social Democrats, said after Sunday's election that they would not join the next government.
Political analysts say that Merkel's party now has to build an uneasy coalition with its nearest rival, the Social Democrats and its candidate, Martin Schulz, coming a distant second with 20.8 per cent of the votes.
However, that's only if she's able to form a coalition with the liberal Free Democrats and the Greens, who finished the day with 10.5 and 9.4 percent of the vote, respectively.
Both parties suffered considerable losses compared to the results of the elections in 2013.
Merkel's victory comes against the backdrop of criticism about her 2015 decision to allow more than a million refugees, mostly people escaping war in the Middle East, into Germany.
The process of trying to form a governing coalition could take months, according to Societe Generale's research team Guy Stear and Brian Hilliard - and could affect the wider direction of the European Union (EU).
"Forming a three-party government is going to be very hard", said Dan Hough, professor of politics at the University of Sussex, who is in Germany to monitor the elections.
She said: 'Of course, we would have preferred a better result, that is completely clear. The coalition is so named because the parties' colors of green, yellow and black mirror the Jamaican flag.
Of the 690 seats, Merkel's party won 238 seats with 32.8% vote share.
Speaking after the vote, Merkel said the success of the far right was a test for Germans and that it was important to listen to the concerns of voters and to win them back.
This is the largest achievement for the party, which will be returning to Germany's parliament after nearly 50 years.
"We have the right to reclaim not just our country, but also our past", he said during a party meeting in September.