From this the wall was continued south to the Birmingham Tower, which is said to have derived its name from its having been erected either by John Birmingham, Earl of Louth, Lord Justice 1321, or Walter Birmingham, who held office in 1348. It is more likely, however, that this tower is identical with that known as the high tower, and that it received the name Birmingham after William Birmingham and his son Walter had been imprisoned there in 1331, otherwise the fortress would have been incomplete prior to 1321, and the side with least natural defence unprotected. From Birmingham Tower the curtain extended eastward (intersected by two smaller towers) to Wardrobe Tower. From this the wall was continued northward to the Store Tower near Dame’s Gate, which was in its turn connected with the eastern gate tower. There were two sallyports in the wall.
Of the eight towers which once protected the fortress only the Wardrobe Tower now remains, often erroneously called Birmingham Tower. This, as its name implies, is where the royal robe, cap of maintenance, and other furniture of state were kept. The sword of state is still preserved there, and also the records which were removed from Birmingham Tower. The lower portion is the original masonry, but an upper storey was added when the Chapel Royal was built. The Birmingham Tower was rendered unsafe by an explosion of gunpowder in Ship Street and had to be taken down. A lighter structure was erected on the site, which contains the present kitchen and viceregal supper-room. The other towers were gradually removed to make room for new buildings.
The Anglo-Irish used to decorate the gate and walls of the castle with the heads of the slain, and in 1316 some four hundred heads were sent from Wicklow for this purpose.
The castle did not become a permanent residence of the Viceroy until 1560 by order of Elizabeth, and in 1565 Henry Sydney took up his abode in it. He also enlarged and beautified it, and placed the State papers there in safe keeping.
The castle has stood several successful sieges. In 1478 Gerald, 8th Earl of Kildare, was superseded as Lord Deputy by Lord Grey of Codnor. He refused to resign, and Keating, Prior of Kilmainham, and Constable of the Castle, sided with him, and fortified the stronghold against the new Deputy. He destroyed the drawbridge, and Lord Grey, finding his numbers too small to force an entrance, returned to England, and Kildare was shortly afterwards reinstated.