The castle is right in front of you and you have plenty of projectiles at your disposal. Hit the structure's weak spots and you will destroy it.