There have been many myths or legends or rumors spreading around about the mysterious castle.
Some say that the Smith's spirit still wandering inside the castle, especially along the corridors, guarding his great mansion. And that's why much of the structure still intact after so many years.
Some say there are lots of "spirits" wandering around the castle since workers died during the construction and people died during the 2nd world war.
It is believed that there are 4 underground tunnels. One is connecting the Kellies Castle to the Hindu temple 500m in the west, one is connecting to the main gate garage in the south and one is connecting to the road in the east.
How about the last one? It's still undiscovered. There are rumors about this secret tunnel had been used as an execution hub of the Japanese army in World War II. And some say it was the secret tunnel being used by Chen Ping (the famous communist leader in Malaya) in between 50s to 60s.
More info : http://www.explore-ipoh.com/kellies-castle.html