Former German president Christian Wulff sparked controversy in 2010 when he said Islam belonged to Germany, a comment repeated by Chancellor Angela Merkel last year.
Six years ago, 49 percent of Germans agreed with Wulff and 47 percent did not.
Thursday’s poll, carried out by Infratest dimap for broadcaster WDR, showed that the mood has shifted, with 60 percent now saying that Islam does not belong to Germany. It showed 34 percent thought it did belong.
Scepticism about the religion was greatest among older people, with 71 percent over the age of 64 believing Islam does not belong to the country.
Germany is home to around four million Muslims, about five percent of the total population, and unease over the religion is on the rise, especially in the wake of deadly attacks in Brussels and Paris by the Daesh (ISIL) terrorist group.
Earlier this month members of the anti-immigration party Alternative for Germany (AfD) backed an election manifesto that says Islam is not compatible with the constitution and calls for a ban on minarets and the burqa.
Just over half of Germans are concerned that the influence of Islam in Germany will become too strong due to the influx of refugees, the Infratest dimap poll showed.
Fears about a terrorist attack in Germany are also rife, with almost three-quarters of Germans worried about the possibility.
The survey of 1,003 Germans was conducted between May 2 and May 3.