Out of all the accounts, the single most consistent detail aside from the old man is the tower's lack of a front door. That might seem inconsequential, but think about it, this tower doesn't have a single entrance nor exit of any kind. During my brief visit the only opening I could find was that window, and it was boarded up no less.
As the old man was yelling at me, I had the foresight to upcast my Detect Magic in order to verify his claim of studying Evocation Magic. It raised more questions, but the existence of the questions at all may bring up some new leads.
See, the aura I felt at the top of the tower was almost purely Enchantment, and the entirety of the tower itself was draped in a sheen of Illusion magic. But I assure you the tower itself was real, before calling out to the man and announcing my presence I checked the walls of the tower by hand for any hidden doors out of suspicion that Illusion magic was at play here. But that's about as far as my knowledge of Illusion goes.
As far as I know there are no official records of this tower nor any writings that reference its existence, only word of mouth. I've decided to call it The Hermit's Tower for now.
Hmm! Quite the mystery...
Indeed!