Migration:  Surfing on the Wave of Technological Evolution

ENSTORE is a very successful petabyte-scale mass storage system developed at Fermilab. Since its inception in the late 1990s, ENSTORE has been serving the Fermilab community, as well as its collaborators, and now holds more than 3.5 petabytes of data on tape. New data is arriving at an ever increasing rate. One practical issue that we are confronted with is: storage technologies have been evolving at an ever faster pace. New drives and media have been brought to the market constantly with larger capacity, better performance, and lower price. It is not cost effective for a forward looking system to stick with older technologies. In order to keep up with this technological evolution, ENSTORE was in need of a mechanism to migrate data onto newer media. Migrating large quantities of data in a highly available mass storage system does present a technical challenge. An auto-migration scheme was developed in ENSTORE that carries out this task seamlessly, behind the scenes, and without interrupting service nor requiring much operational attention. After two years in service, auto-migration has lived up to its expectations and ENSTORE has gone through several generations of drives and media. In addition, migration can be used in media copying, media consolidation, and data compaction. In this paper, we are going to present the conceptual design of ENSTORE, the issues in data migration in a highly available mass storage system, the implementation of auto-migration in ENSTORE, our experience and its extended applications