ElasticSearch Index for DDI 3.2 Studies and Variables. Explore-Data project

DDI 3.2 presents a solid meta-data model to encode social sciences surveys meta-data, such as studies and variables. However, making this data accessible is another story. In Explore-Data project, our goal is to present a dynamic, multi-facets, and multilingual search portal for both studies and variables. Accordingly, we transform the content of DDI, available via DDI-FlatDB, into ElasticSearch index. We consider two main document types in our index, namely study and variable. Each document type consists of a set of fields in order to keep as much of study/variable structure as possible. To support the multilingual content, we build many copies of each field for each language. During the indexing process, we were obliged to deal with migration of data, missing values, remove duplicated and redundant data, and sometimes to generate new data. Index content can be queried via classical Google-style queries. However, the same content can be also browsed via many pre-defined facets in an e-Commerce websites style. Both ways can be used simultaneously in order to give to users a tool to dynamically access data and filter result list. We have indexed about 6000 studies, distributed on 13 study collections such as ALLBUS, European Values Study, etc. In addition we currently index about 150K variables. In our presentation we introduce the Explore Data project at GESIS and describe indexing challenges to transform DDI Lifecycle 3.2 to ElasticSearch on study and variable level.