Simply because most people see themselves as aside from issues around them, they fall short to see their nature as component of them.
The rest of the castle's grounds are stated to have other spirit vacationers as well.