The need for some University building was, however, severely felt. At last it was provided for in a small way. "That memorable fabric, the old Congregation House," and the room above it were begun in 1320 by the above-mentioned Adam de Brome, at the expense of Thomas Cobham, Bishop of Worcester. The latter had undertaken to enlarge the old fabric of S. Mary's Church by erecting a building two stories high immediately to the east of the tower, on the very site, that is, on which the University had previously endeavoured to found a chantry. He intended that the lower room should serve primarily as a meeting-place for the Congregations of regent-masters, and at other times for parochial purposes. The upper room was to be used partly as an oratory, and partly as a general library. But the good bishop's books, which were to form the nucleus of this library, met with the same fate as Richard de Bury's. His executors pawned them to defray the expenses of his funeral, and to pay his debts. Oriel College at their suggestion redeemed the books, and being also the impropriating rectors of the church, they claimed to treat both building and library as their own property. But the masters presently asserted their supposed rights by coming "with a great multitude" and forcibly carrying away the books from Oriel, "in autumn, when the fellows were mostly away." They lodged the books in the upper chamber, and Oriel presently acknowledged the University's proprietary rights.
The old University library, then, found its home in the upper room of the old Congregation House, and there remained until the books were moved to Duke Humphrey's library (1480). From that time till the erection of Laud's Convocation House, the upper room was used as a school of law, and also as another Congregation House, distinguished by the name of "Upper." Meantime a salary was provided for a librarian, who, besides taking care of the books in the upper chamber, was to pray for the soul of the donors. Other books were acquired by the University, either by purchase, bequest, or as unredeemed pledges. Some of these were kept in chests, and loaned out on security like cash from the other chests, whilst others were books given or bequeathed to the University, which were kept chained in the chancel of the church, where the students might read them. Others, in the upper room, were secured to shelves by chains that ran on iron bars. These shelves, with desks alongside, would run out from the walls, between the seven windows, in a manner clearly shown by such survivals of mediæval libraries as exist at the Bodleian, Merton and Corpus. The catalogue was in the form of a large board suspended in the room. At first these books were open for the use of all students at the specified times, but by later statutes (1412), made when the library had been increased by further donations and time had brought bitter experience, the use of them was stringently limited to graduates or religious of eight years' standing in Philosophia. These regulations were intended to provide against the overcrowding of the small library, the disturbance of readers and the destruction of books by careless, idle and not over-clean boy students. With the object of preserving the books, a solemn oath was also exacted from all graduates on admission to their degree, that they would use them well and carefully. The lower room fulfilled its founder's intention, and here the Congregation of regents met, whilst the Convocation, or Great Congregation of regents and non-regents, was held in the chancel of the church.