From Hydras to TACOs

Stanford University Library has a robust digital library system called the Stanford Digital Repository. This repository holds a little under 500 TB of materials in preservation and online for researchers, capture of scholarly output, and digitized cultural heritage materials. These materials are managed across 90+ codebases serving a variety of functions from self-deposit web applications, to a nearly 10 year old parallel processing framework, to a digital repository assets publication mechanism leading into our Blacklight, Spotlight, and Geoblacklight applications – among other services and needs. At the core of this system is a Fedora 3 store. With Fedora 3 now end-of-lifed, and our system suffering from limited to no horizontal scalability options, we’re revisiting our system and architecture. We are writing it from the start with a goal to have data-forward, distributed microservices and some event-driven processing components. TACO, our new core management API, is the heart of this new architecture, and is currently being developed as a prototype. This talk will walk through the process of analysing our current system via a dataflows analysis; designing a new architecture for our digital library with a wide ranging set of requirements and users; prototyping a core component of our new architecture to be horizontally scalable as well as data & specification driven; then planning how to create ‘seams’ in our current system to migrate towards our new system in an evolutionary fashion instead of a turn-key migration