Providing Access to a Data Library: SQL and Full-Text IR Methods of Automatically Generating Web Structure

Social Science research in universities has traditionally been supported by a central data archive, holding both the raw data (e.g. the U.S. census) and the wide variety of support materials necessary to identify, understand and manipulate the raw data. Columbia University's Electronic Data Service (EDS) has built an online system using World-Wide-Web technology to offer these support materials, and ultimately the raw data itself, to our research community. We describe our historical motivations, our data format difficulties, our filing systems and our scalable technology solution. Our emphasis is on a new set of software, inter-connected by Web protocols, which is easy to use and is self-maintaining