Well, being a god can affect his actions subconsciously, you know. Completely believing that a course of action would unfold in the way he sees it happening would rewrite reality into making such a thing possible if he were to try it, while he still isn't using his god powers on command.
Which is the technical explanation, which ruins the magic.
But come on, a little impossibility would be fun. It probably wouldn't even take that much time to fix considering how fast you're building the castle, so just let me wreck a tiny bit of it.
Though technically, I don't really need your permission. I'm just being polite.