It is uncertain whether any of the Norman kings before Stephen made the Tower a place of residence. But in 1140 that monarch, during a gloomy period of private and public affairs, retired to the Tower with a large retinue and kept his court there during Whitsuntide.
“Early in the year,” writes Freeman in his “History of the Norman Conquest,” “after Matilda’s landing, an attempt had been made to make peace. At Pentecost the King held, or tried to hold, the usual festival in London; but this time his court was held to the east and not to the west of the city, not in the hall of Rufus, but in the fortress of his father.”
The custody of the Tower appears, soon after its completion, to have been made an hereditary office, granted by the sovereign to the family of Mandeville. In this year of 1140 the Tower was in the keeping of Geoffrey, grandson of that great Geoffrey de Mandeville, who had accompanied the Conqueror to England, and who had greatly distinguished himself at the Battle of Hastings. Stephen created the grandson Earl of Essex, but being himself taken prisoner soon afterwards at the Battle of Lincoln, the Empress Matilda gained de Mandeville over to her party, during Stephen’s captivity. By a charter, dated from Oxford in 1141, Matilda confirmed the Earl in all the possessions which he had inherited, whether in lands or fortresses, the custody of the Tower being included therein, Essex being given a free hand to strengthen and fortify it. A subsequent charter of the same year gave him the special charge of the Tower, “with all lands, liveries, and customs thereto appertaining” (Dugdale’s Baronage). According to Leland, de Mandeville constantly added to the fortifications of the Tower, but when he was defeated and taken prisoner at the Battle of St Albans he was obliged to surrender the Constableship into the hands of Stephen.
In 1153 the Tower was held for the Crown by Richard de Lucy, Chief Justiciary of England, in trust for Henry, Duke of Normandy, to whom, after Stephen’s death, it reverted.
Matilda had offended the Londoners by refusing to abolish her father’s laws, and by also refusing to restore those granted by Edward the Confessor, and, rising in arms, they drove the Empress from the city. Stephen having recovered his liberty, Matilda’s power ceased shortly afterwards. After her flight the Londoners laid siege to the Tower, but it had been so strongly fortified by de Mandeville that he was not only able to defy the besiegers’ uttermost efforts to effect its capture, but was able to make a sortie as far as Fulham, where he took the Bishop of London prisoner, “as then lodged there, being of the contrary faction” (Holinshed).