Exploiting Data Parallelism in the Image Content Engine

The Image Content Engine (ICE) is a framework of software and underlying mathematical and physical models that enable scientists and analysts to extract features from Terabytes of imagery and search the extracted features for content relevant to their problem domain. The ICE team has developed a set of tools for feature extraction and analysis of image data, primarily based on the image content. The scale and volume of imagery that must be searched presents a formidable computation and data bandwidth challenge, and a search of moderate to large scale imagery quickly becomes intractable without exploiting high degrees of data parallelism in the feature extraction engine. In this paper we describe the software and hardware architecture developed to build a data parallel processing engine for ICE. We discuss our highly tunable parallel process and job scheduling subsystem, remote procedure invocation, parallel I/O strategy, and our experience in running ICE on a 16 node, 32 processing element (CPU) Linux Cluster. We present performance and benchmark results, and describe how we obtain excellent speedup for the imagery searches in our test-bed prototype