Verse 37 of Surah Ibrahim notes that when Hazrat Ibrahim (AS) arrived in Mecca with his wife and son, the Kaaba existed. “Lord, I have settled some of my offspring in a barren valley near your Sacred House so that they could be steadfast in prayer.”
There is another reference to Kaaba in verse 36 of Surah Al Imran: “The first house (of worship) that God assigned to men was in Bakka (another name of Mecca). It is a blessed one and a guide for all people.”
Surely, worshiping God and constructing buildings for this purpose did not start at the time of Ibrahim (AS), rather, they can be traced back to even the time of Adam (AS). This shows that the foundations of the Kaaba had already existed and Hazrat Ibrahim and his son Ismail have built the structure upon it.
Meanwhile, Imam Ali (AS) also points to the topic in a sermon. “Don’t you see that God has tested people from the time of Adam to now by stones? God has made this His house and ordered Adam and his children to do tawaf around that.”
According to the Quran and hadith, the Kaaba was first built by Adam (AS) but was later demolished during Nuh’s flood before being reconstructed by Ibrahim (AS) and his son.