Stochastic control optimization & simulation applied to energy management: From 1-D to N-D problem distributions, on clusters, supercomputers and Grids

International audienceManagement of electricity production to control cost while satisfying demand, leads to solve a stochastic optimization problem where the main sources of uncertainty are the demand load, the electricity and fuel market prices, the hydraulicity, and the availability of the thermal production assets. A stochastic dynamic programming method is an interesting solution for non convex optimization, but is both CPU and memory consuming. It requires parallelization to achieve speedup and size up, and to deal with a big number of stocks (N) and a big number of uncertainty factors. This talk will introduce a collaboration between EDF (a French electricity producer) and SUPELEC (a French engineering school and research laboratory) that aimed to distribute N-dimension stochastic dynamic programming applications on large distributed architectures, like PC clusters and IBM Blue Gene supercomputers. This collaboration was initiated in a French ANR project about Distributed and Grid computing applied to financial mathematic problems (the “GCPMF” ANR project). From an applicative point of view, the goal of this research was to be able to deal with at least three or four uncertainty factors, and at least six or seven stocks in optimization, while being able to efficiently use in simulation the commands calculated. The simulations are used after optimization in order to generate gain estimations on different periods and in order to estimate the associated risks. The methodology developed in this research project will bring some reference calculations that will help to derive some simplified versions to use in production. From a computer science point of view, three different parallelization strategies have been carried out in order to access input and output files from thousands of processors, to distribute a N dimensional cube of data used at each time step of an optimization algorithm, and to compute independent simulations requiring data spread in many separate files managed by different processors. All designed parallel algorithms have been experimented on a 7- stocks problem (7-dimensions problem) on different parallel architectures. We successfully used up to 256 processors of a PC cluster and up to 8192 processors of a Blue Gene/L supercomputer, achieving scalability with regular decrease of the execution time. We started distributing a 1-dimension stochastic control algorithm (applied to a gas storage valuation) in February 2007, and we extended our distribution to a N- dimension algorithm in 2008 (applied to electricity production management). In the next months this industrial and large scale distributed application will be used: – by EDF to study and optimize its energetic stock management and electricity production, using its new Blue Gene/P supercomputer up to 32000 processors; – by SUPELEC (IMS group) and INRIA (AlGorille and Reso teams) to run large experiments on Grid'5000, analyze communications and performances, and optimize task distribution when using several sites of Grid'5000. A global collaboration between EDF, SUPELEC and INRIA will allow comparing performances of this real and not embarrassingly parallel application, on supercomputers, different large PC-clusters and one multi-site Grid