Taaffe's Castle is a fortified town house near the harbour, its main tower built in the early 16th century and a side extension added later. Local tradition links it to the Taaffe family, who became Earls of Carlingford in 1661, but there is no evidence they built or owned it, and early maps show it was already standing before they got their title.
Its position close to the old harbour suggests a trading depot on the ground floor with living quarters above, a reminder that Carlingford's wealth came from the sea.