Dreaming machine learning: Lipschitz extensions for reinforcement learning on financial markets

[EN] We consider a quasi-metric topological structure for the construction of a new reinforcement learning model in the framework of financial markets. It is based on a Lipschitz type extension of reward functions defined in metric spaces. Specifically, the McShane and Whitney extensions are considered for a reward function which is defined by the total evaluation of the benefits produced by the investment decision at a given time. We define the metric as a linear combination of a Euclidean distance and an angular metric component. All information about the evolution of the system from the beginning of the time interval is used to support the extension of the reward function, but in addition this data set is enriched by adding some artificially produced states. Thus, the main novelty of our method is the way we produce more states-which we call "dreams"-to enrich learning. Using some known states of the dynamical system that represents the evolution of the financial market, we use our technique to simulate new states by interpolating real states and introducing some random variables. These new states are used to feed a learning algorithm designed to improve the investment strategy by following a typical reinforcement learning scheme. (C) 2020 Elsevier B.V. All rights reserved.This work was supported by the Ministerio de Ciencia, Innovacion y Universidades, Agencial Estatal de Investigaciones and FEDER (Spain) (grant number MTM2016-77054-C2-1-P.)Calabuig, JM.; Falciani, H.; Sánchez Pérez, EA. (2020). Dreaming machine learning: Lipschitz extensions for reinforcement learning on financial markets. Neurocomputing. 398:172-184. https://doi.org/10.1016/j.neucom.2020.02.052172184398Aliprantis, C., & Burkinshaw, O. (2003). Locally Solid Riesz Spaces with Applications to Economics. Mathematical Surveys and Monographs. doi:10.1090/surv/105Almahdi, S., & Yang, S. Y. (2017). An adaptive portfolio trading system: A risk-return portfolio optimization using recurrent reinforcement learning with expected maximum drawdown. Expert Systems with Applications, 87, 267-279. doi:10.1016/j.eswa.2017.06.023Aronsson, G. (1967). Extension of functions satisfying lipschitz conditions. Arkiv för Matematik, 6(6), 551-561. doi:10.1007/bf02591928Bekiros, S. D. (2010). Heterogeneous trading strategies with adaptive fuzzy Actor–Critic reinforcement learning: A behavioral approach. Journal of Economic Dynamics and Control, 34(6), 1153-1170. doi:10.1016/j.jedc.2010.01.015Bekiros, S. D. (2015). Heuristic learning in intraday trading under uncertainty. Journal of Empirical Finance, 30, 34-49. doi:10.1016/j.jempfin.2014.11.002Bertoluzzo, F., & Corazza, M. (2012). Testing Different Reinforcement Learning Configurations for Financial Trading: Introduction and Applications. Procedia Economics and Finance, 3, 68-77. doi:10.1016/s2212-5671(12)00122-0Cavalcante, R. C., Brasileiro, R. C., Souza, V. L. F., Nobrega, J. P., & Oliveira, A. L. I. (2016). Computational Intelligence and Financial Markets: A Survey and Future Directions. Expert Systems with Applications, 55, 194-211. doi:10.1016/j.eswa.2016.02.006Chen, Y., & Hao, Y. (2017). A feature weighted support vector machine and K-nearest neighbor algorithm for stock market indices prediction. Expert Systems with Applications, 80, 340-355. doi:10.1016/j.eswa.2017.02.044Chong, E., Han, C., & Park, F. C. (2017). Deep learning networks for stock market analysis and prediction: Methodology, data representations, and case studies. Expert Systems with Applications, 83, 187-205. doi:10.1016/j.eswa.2017.04.030Das, S. P., & Padhy, S. (2015). A novel hybrid model using teaching–learning-based optimization and a support vector machine for commodity futures index forecasting. International Journal of Machine Learning and Cybernetics, 9(1), 97-111. doi:10.1007/s13042-015-0359-0Defoort, M., Polyakov, A., Demesure, G., Djemai, M., & Veluvolu, K. (2015). Leader‐follower fixed‐time consensus for multi‐agent systems with unknown non‐linear inherent dynamics. IET Control Theory & Applications, 9(14), 2165-2170. doi:10.1049/iet-cta.2014.1301Dempster, M. A. H., & Leemans, V. (2006). An automated FX trading system using adaptive reinforcement learning. Expert Systems with Applications, 30(3), 543-552. doi:10.1016/j.eswa.2005.10.012Deng, Y., Bao, F., Kong, Y., Ren, Z., & Dai, Q. (2017). Deep Direct Reinforcement Learning for Financial Signal Representation and Trading. IEEE Transactions on Neural Networks and Learning Systems, 28(3), 653-664. doi:10.1109/tnnls.2016.2522401M. Dong, X. Yang, Y. Wu, J.H. Xue, Metric learning via maximizing the Lipschitz margin ratio, arXiv:1802.03464 (2018) 1–12.Driessens, K., Ramon, J., & Gärtner, T. (2006). Graph kernels and Gaussian processes for relational reinforcement learning. Machine Learning, 64(1-3), 91-119. doi:10.1007/s10994-006-8258-yDunis, C. L., Rosillo, R., de la Fuente, D., & Pino, R. (2012). Forecasting IBEX-35 moves using support vector machines. Neural Computing and Applications, 23(1), 229-236. doi:10.1007/s00521-012-0821-9Fischer, T., & Krauss, C. (2018). Deep learning with long short-term memory networks for financial market predictions. European Journal of Operational Research, 270(2), 654-669. doi:10.1016/j.ejor.2017.11.054Gerlein, E. A., McGinnity, M., Belatreche, A., & Coleman, S. (2016). Evaluating machine learning classification for financial trading: An empirical approach. Expert Systems with Applications, 54, 193-207. doi:10.1016/j.eswa.2016.01.018Gottlieb, L.-A., Kontorovich, A., & Krauthgamer, R. (2014). Efficient Classification for Metric Data. IEEE Transactions on Information Theory, 60(9), 5750-5759. doi:10.1109/tit.2014.2339840Guo, X.-G., Wang, J. L., Liao, F., & Teo, R. S. H. (2016). Distributed adaptive control for vehicular platoon with unknown dead-zone inputs and velocity/acceleration disturbances. International Journal of Robust and Nonlinear Control, 27(16), 2961-2981. doi:10.1002/rnc.3720Jeong, G., & Kim, H. Y. (2019). Improving financial trading decisions using deep Q-learning: Predicting the number of shares, action strategies, and transfer learning. Expert Systems with Applications, 117, 125-138. doi:10.1016/j.eswa.2018.09.036Kearney, C., & Liu, S. (2014). Textual sentiment in finance: A survey of methods and models. International Review of Financial Analysis, 33, 171-185. doi:10.1016/j.irfa.2014.02.006Lahmiri, S. (2016). A variational mode decompoisition approach for analysis and forecasting of economic and financial time series. Expert Systems with Applications, 55, 268-273. doi:10.1016/j.eswa.2016.02.025Lee, T. K., Cho, J. H., Kwon, D. S., & Sohn, S. Y. (2019). Global stock market investment strategies based on financial network indicators using machine learning techniques. Expert Systems with Applications, 117, 228-242. doi:10.1016/j.eswa.2018.09.005Lee, J. W., Park, J., O, J., Lee, J., & Hong, E. (2007). A Multiagent Approach to $Q$-Learning for Daily Stock Trading. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, 37(6), 864-877. doi:10.1109/tsmca.2007.904825Li, Y., Jiang, W., Yang, L., & Wu, T. (2018). On neural networks and learning systems for business computing. Neurocomputing, 275, 1150-1159. doi:10.1016/j.neucom.2017.09.054Liu, F., & Wang, J. (2012). Fluctuation prediction of stock market index by Legendre neural network with random time strength function. Neurocomputing, 83, 12-21. doi:10.1016/j.neucom.2011.09.033LOUGHRAN, T., & MCDONALD, B. (2016). Textual Analysis in Accounting and Finance: A Survey. Journal of Accounting Research, 54(4), 1187-1230. doi:10.1111/1475-679x.12123Lu, C.-J., Lee, T.-S., & Chiu, C.-C. (2009). Financial time series forecasting using independent component analysis and support vector regression. Decision Support Systems, 47(2), 115-125. doi:10.1016/j.dss.2009.02.001Mahmoudi, N., Docherty, P., & Moscato, P. (2018). Deep neural networks understand investors better. Decision Support Systems, 112, 23-34. doi:10.1016/j.dss.2018.06.002Maringer, D., & Ramtohul, T. (2011). Regime-switching recurrent reinforcement learning for investment decision making. Computational Management Science, 9(1), 89-107. doi:10.1007/s10287-011-0131-1McShane, E. J. (1934). Extension of range of functions. Bulletin of the American Mathematical Society, 40(12), 837-842. doi:10.1090/s0002-9904-1934-05978-0Milman, V. A. (1999). Absolutely minimal extensions of functions on metric spaces. Sbornik: Mathematics, 190(6), 859-885. doi:10.1070/sm1999v190n06abeh000409Moghaddam, A. H., Moghaddam, M. H., & Esfandyari, M. (2016). Stock market index prediction using artificial neural network. Journal of Economics, Finance and Administrative Science, 21(41), 89-93. doi:10.1016/j.jefas.2016.07.002Moody, J., & Saffell, M. (2001). Learning to trade via direct reinforcement. IEEE Transactions on Neural Networks, 12(4), 875-889. doi:10.1109/72.935097Khadjeh Nassirtoussi, A., Aghabozorgi, S., Ying Wah, T., & Ngo, D. C. L. (2014). Text mining for market prediction: A systematic review. Expert Systems with Applications, 41(16), 7653-7670. doi:10.1016/j.eswa.2014.06.009H. Park, M.K. Sim, D.G. Choi, An intelligent financial portfolio trading strategy using deep q-learning, arXiv:1907.03665 (2019) 1–39.Patel, J., Shah, S., Thakkar, P., & Kotecha, K. (2015). Predicting stock market index using fusion of machine learning techniques. Expert Systems with Applications, 42(4), 2162-2172. doi:10.1016/j.eswa.2014.10.031Pendharkar, P. C., & Cusatis, P. (2018). Trading financial indices with reinforcement learning agents. Expert Systems with Applications, 103, 1-13. doi:10.1016/j.eswa.2018.02.032Romaguera, S., & Sanchis, M. (2000). Semi-Lipschitz Functions and Best Approximation in Quasi-Metric Spaces. Journal of Approximation Theory, 103(2), 292-301. doi:10.1006/jath.1999.3439Schmidhuber, J. (2015). Deep learning in neural networks: An overview. Neural Networks, 61, 85-117. doi:10.1016/j.neunet.2014.09.003Sezer, O. B., & Ozbayoglu, A. M. (2018). Algorithmic financial trading with deep convolutional neural networks: Time series to image conversion approach. Applied Soft Computing, 70, 525-538. doi:10.1016/j.asoc.2018.04.024Tkáč, M., & Verner, R. (2016). Artificial neural networks in business: Two decades of research. Applied Soft Computing, 38, 788-804. doi:10.1016/j.asoc.2015.09.040Ticknor, J. L. (2013). A Bayesian regularized artificial neural network for stock market forecasting. Expert Systems with Applications, 40(14), 5501-5506. doi:10.1016/j.eswa.2013.04.013Wang, B., Huang, H., & Wang, X. (2012). A novel text mining approach to financial time series forecasting. Neurocomputing, 83, 136-145. doi:10.1016/j.neucom.2011.12.013Wang, B., Huang, H., & Wang, X. (2011). A support vector machine based MSM model for financial short-term volatility forecasting. Neural Computing and Applications, 22(1), 21-28. doi:10.1007/s00521-011-0742-zXiao, G., Zhang, H., Luo, Y., & Qu, Q. (2017). General value iteration based reinforcement learning for solving optimal tracking control problem of continuous–time affine nonlinear systems. Neurocomputing, 245, 114-123. doi:10.1016/j.neucom.2017.03.038Yeh, C.-Y., Huang, C.-W., & Lee, S.-J. (2011). A multiple-kernel support vector regression approach for stock market price forecasting. Expert Systems with Applications, 38(3), 2177-2186. doi:10.1016/j.eswa.2010.08.004Zhang, X., Hu, Y., Xie, K., Zhang, W., Su, L., & Liu, M. (2015). An evolutionary trend reversion model for stock trading rule discovery. Knowledge-Based Systems, 79, 27-35. doi:10.1016/j.knosys.2014.08.010Zhang, J., & Maringer, D. (2015). Using a Genetic Algorithm to Improve Recurrent Reinforcement Learning for Equity Trading. Computational Economics, 47(4), 551-567. doi:10.1007/s10614-015-9490-yZhiqiang, G., Huaiqing, W., & Quan, L. (2012). Financial time series forecasting using LPP and SVM optimized by PSO. Soft Computing, 17(5), 805-818. doi:10.1007/s00500-012-0953-