The source of the curse was located on an island not far off the keeps shore. But it's inhabited by more than just the witch. A cult entrenched in the slave trade made a place for themselves. Once the island had been cleared and the curse broken they were able to return to the keep and start restoration.