The grottos, which, according to the fashion of the time, were built in the garden, and richly decorated, must have seen some interesting sights. One in which Queen Caroline was royally entertained in 1729 was taken down in 1795. The entertainment was extremely sumptuous. The last of these grottos disappeared only when the Embankment was being made. In 1741 the Minister retired with the title of Earl of Orford, which afterwards descended to his well-known son, Horace, and a pension of £4,000 a year.
The house afterwards passed through the hands of John, Earl of Dunmore, and George Aufrere, and we find it in 1796 assigned to Charles, Lord Yarborough, who was living here in 1808. The building being then required by the Hospital, he consented to give up the remainder of his lease, a period of seventeen years, upon compensation being paid to the amount of £4,775 15s. Sir John Soane, the architect, who had all through been strongly in favour of adding on to Walpole House instead of purchasing new ground, designed the necessary additions. The building, like the Hospital itself, consists of two wings, east and west, abutting out from a connecting flank, with a vestibule in the front. The eastern wing is Walpole House. The room which was originally the dining-room is now one of the wards, and contains eight beds. It is strange to see the worn, homely faces of the infirm pensioners, in contrast with the magnificent white marble mantelpiece and the finely moulded ceiling. The connecting wing holds the Matron's room in addition to the wards. The patients suffer from the complaints of old age, rheumatism, blindness, paralysis; few of them are permanently in the infirmary, and with the season of the year the numbers vary. In the summer it is found possible to close one ward entirely. There is a staff of nurses, and the old men are well looked after. Besides Walpole House, it was considered advisable to have a supplementary infirmary. So when the lease of Gordon House fell in, it was adapted for the purpose. It stands in the southwest corner of the grounds, about 150 yards from the infirmary, and will be familiar to those who visited the Military and Naval Exhibitions, at which period it was used as a refreshment-house. The first recorded lease of the land on which it was built was in 1690.