The gallery stands on the site of the old Millbank Penitentiary, for the scheme of which Howard the reformer was originally responsible. He was annoyed by the rejection of the site he advocated, however, and afterwards withdrew from the project altogether. Wandsworth Fields and Battersea Rise were both discussed as possible sites, but were eventually abandoned in favour of Millbank. Jeremy Bentham, who advocated new methods in the treatment of prisoners, gained a contract from the Government for the erection and management of the new prison. He, however, greatly exceeded the terms of his contract, and finally withdrew, and supervisors were appointed. The prison was a six-rayed building with a chapel in the centre. Each ray was pentagonal in shape, and had three towers on its exterior angles. The whole was surrounded by an octagonal wall overlooking a moat. At the closing of the prison in Tothill Fields it became the sole Metropolitan prison for females, "just as," says Major Griffiths, "it was the sole reformatory for promising criminals, the first receptacle for military prisoners, the great depot for convicts _en route_ for the antipodes."
In 1843 it was called a penitentiary instead of a prison. Gradually, as new methods of prison architecture were evolved, Millbank was recognised as cumbersome and inadequate. It was doomed for many years before its demolition, and now, like the prison of Tothill Fields, has vanished. Even the convicts' burial-ground at the back of the Tate Gallery is nearly covered with County Council industrial dwellings.
Further northward in the Grosvenor Road, Peterborough House once stood, facing the river, and this was at one time called "the last house in Westminster." It was built by the first Earl of Peterborough, and retained his name until 1735, when it passed to Alexander Davis of Ebury, whose only daughter and heiress had married Sir Thomas Grosvenor. It was by this marriage that the great London property came into the possession of the Grosvenor (Westminster) family. The house was rebuilt, and renamed Grosvenor House. Strype says: "The Earl of Peterborough's house with a large courtyard before it, and a fine garden behind, but its situation is but bleak in winter and not over healthful, as being too near the low meadows on the south and west parts." The house was finally demolished in 1809.