High Performance Risk Aggregation: Addressing the Data Processing Challenge the Hadoop MapReduce Way

Monte Carlo simulations employed for the analysis of portfo-lios of catastrophic risk process large volumes of data. Often times these simulations are not performed in real-time sce-narios as they are slow and consume large data. Such sim-ulations can benefit from a framework that exploits paral-lelism for addressing the computational challenge and facil-itates a distributed file system for addressing the data chal-lenge. To this end, the Apache Hadoop framework is chosen for the simulation reported in this paper so that the com-putational challenge can be tackled using the MapReduce model and the data challenge can be addressed using the Hadoop Distributed File System. A parallel algorithm for the analysis of aggregate risk is proposed and implemented using the MapReduce model in this paper. An evaluation of the performance of the algorithm indicates that the Hadoop MapReduce model offers a framework for processing large data in aggregate risk analysis. A simulation of aggregate risk employing 100,000 trials with 1000 catastrophic events per trial on a typical exposure set and contract structure is performed on multiple worker nodes in less than 6 minutes. The result indicates the scope and feasibility of MapReduce for tackling the computational and data challenge in the analysis of aggregate risk for real-time use. Keywords hadoop mapreduce; risk aggregation; risk analysis; data pro-cessing; high-performance analytics 1