I think OP meant ancient to medieval and not industrial 1861-1865

Castles have some iron fittings e.g. portcullis spikes, sconces etc. In order to plate wall with metal that could withstand e.g. a trebuchet or ballista you're talking battleship thickness. Stone and timber were much cheaper and easier to transport for repairs. Considering as well a suit of armour would have cost $100,000 in today's money because metal wasn't mass produced it wouldn't have been economically viable to plate a building and good luck finding enough ore without modern mining methods.