1Collection Manager: Integrating Diverse Data Sources on the Grid

Abstract: Today’s grid applications face numerous challenges associated with aggregation of diverse distributed data sources and types available across distributed domains. GridIR, a grid based information retrieval system faces similar challenges. A key component of the GridIR model is the Collection Manager (CM) that builds and publishes uniform data collections populated with data from heterogeneous sources both on and off the grid. CM facilitates different data sources to be grouped together as a collection and helps monitor for changes consequently. Collection Manager was designed as part of the Grid Information Retrieval System, to manage dispersed heterogeneous data sources. However, CM can work with any other system that needs data access and management in a grid environment. CM is built on the Open Grid Service Architecture (OGSA) and thus takes advantage of security and notification mechanisms available within the architecture. This paper describes the architecture of the Collection Manager and describes how extensions can be developed to make proprietary data sources accessible on the grid. 1