By the year 1725, sadly, the castle was not looked after and had fallen into ruin. It was then used as a market place for the townspeople and later as a courthouse. At one time, theives who terrorised the country were bound in stocks in the castle for a long time.Things got from bad to worse, in 1736, the castle was struck by a bolt of lighting destoying many of the turrets and in 1737, very high winds blew most of the roof off. The castle gates were eventually removed and taken to the Hamilton family's burial vaults.
As time went on, walls collapsed or were pushed down for stone to build houses and eventually, the castle got so bad, it had to be secured and was closed for a long time so that the council could make it safe. Unfortunatley, for the castle to stay standing, the inner curtain wall had to be covered with concrete (in 1912), but is not visable in the castle's picture-postcard postion.
The Castle ruins, according to local legend, is home to the ghost of the White Lady. The legend says, that one of the Lord's wife's displeased her husband so badly, he had her put into a small chamber, given some food, blessed by a priest and walled up forever. When some walls fell down in the late 19th century, human bones were found, giving some evidence to the story.