E.J.: Jaql: A scripting language for large scale semistructured data analysis

Jaql is a declarative scripting language for enterprise data analysis powered by a scalable runtime that leverages Hadoop’s MapReduce parallel programming framework. Jaql is used in IBM’s Cognos Consumer Insight [6], the announced InfoSphere BigInsights [3], as well as several research projects. Through these interactions and use-cases, we have focused on the following core design prin-ciples: (1) a flexible data model, (2) reusability, (3) varying lev-els of abstraction, and (4) scalability. The data model is inspired by JSON and can be used to represent data that varies from flat, relational tables to collections of semi-structured documents. To support the various phases of data analysis, Jaql is able to operate without knowledge of the schema of the data; queries can evolve towards partial or rigid schema definitions over time. Reusability is provided through the use of higher-order functions, and by pack-aging related functions and their required resources into modules. While Jaql is declarative, it is built from layers that vary in their level of abstraction. Higher levels allow concise specification of logical operations (e.g., join), while lower levels blend in physi-cal aspects (e.g., hash join or MapReduce job). We have exposed such functionality so that it is easier to add new operators or to pin down an execution plan for greater control. Finally, Jaql auto-matically rewrites scripts to use Hadoop’s MapReduce for parallel data processing, when possible. Our experimental results illustrate that Jaql scales well for real and synthetic workloads and highlights how access to lower-level operators enabled us to parallelize typi-cally sequential flows for scalably analyzing large datasets. 1