No one knows when the first duergar came to the Underdark below the Vusclands, but it must have been a very long time ago. Their entire culture was centered on a set of fortresses and cities below (what is now) Vul Vusc, although it was spread over (or under) the entire island. Their capital city/fortress was Thoduum.
They never ruled on the surface, but that is probably because on the surface there was really no one to rule at the time (~1000 years ago). It is also not known what eventually caused the empire to fall, but it is widely assumed that therise of the goddess Deep Duerra and the fractious battles between her loyal followers and the followers of Laduguer, and the generally fractious nature of duergar, made it inevitable.
One particularly notable aspect of duergar architecture is the duergar puzzle wall.