Neural networks and reinforcement learning in wind turbine control

[EN] Pitch control of wind turbines is complex due to the intrinsic non-linear behavior of these devices, and the external disturbances they are subjected to related to changing wind conditions and other meteorological phenomena. This difficulty is even higher in the case of floating offshore turbines, due to ocean currents and waves. Neural networks and other intelligent control techniques have been proven very useful for the modeling and control of these complex systems. Thus, this paper presents different intelligent control configurations applied to wind turbine pitch control. Direct pitch control based on neural networks and reinforcement learning, and some hybrid control configurations are described. The usefulness of neuro-estimators for the improvement of controllers is also presented. Finally, some of these techniques are used in an application example with a land wind turbine model.[ES] El control del ángulo de las palas de las turbinas eólicas es complejo debido al comportamiento no lineal de los aerogeneradores, y a las perturbaciones externas a las que están sometidas debido a las condiciones cambiantes del viento y otros fenómenos meteorológicos. Esta dificultad se agrava en el caso de las turbinas flotantes marinas, donde también les afectan las corrientes marinas y las olas. Las redes neuronales, y otras técnicas del control inteligente, han demostrado ser muy útiles para el modelado y control de estos sistemas. En este trabajo se presentan diferentes configuraciones de control inteligente, basadas principalmente en redes neuronales y aprendizaje por refuerzo, aplicadas al control de las turbinas eólicas. Se describe el control directo del ángulo de las palas del aerogenerador y algunas configuraciones híbridas de control. Se expone la utilidad de los neuro-estimadores para la mejora de los controladores. Finalmente, se muestra un ejemplo de aplicación de algunas de estas técnicas en un modelo de turbina terrestre.Ministerio de Ciencia, Innovación y Universidades: Proyecto MCI AEI/FEDER RTI2018- 094902-B-C21Sierra-García, JE.; Santos, M. (2021). Redes neuronales y aprendizaje por refuerzo en el control de turbinas eólicas. Revista Iberoamericana de Automática e Informática industrial. 18(4):327-335. https://doi.org/10.4995/riai.2021.16111327335184Abouheaf, M., Gueaieb, W., Sharaf, A. 2018. Model-free adaptive learning control scheme for wind turbines with doubly fed induction generators. IET Renewable Power Generation 12(14), 1675-1686. https://doi.org/10.1049/iet-rpg.2018.5353Alvarez-Ramos, C. M., Santos, M., López, V. 2010. Reinforcement learning vs. A* in a role playing game benchmark scenario. In Computational Intelligence: Foundations and Applications (pp. 644-650). https://doi.org/10.1142/9789814324700_0097Asghar, A. B., Liu, X. 2018a. Adaptive neuro-fuzzy algorithm to estimate effective wind speed and optimal rotor speed for variable-speed wind turbine. Neurocomputing 272, 495-504. https://doi.org/10.1016/j.neucom.2017.07.022Asghar, A. B., Liu, X. 2018b. Estimation of wind speed probability distribution and wind energy potential using adaptive neuro-fuzzy methodology. Neurocomputing, 287, 58-67. https://doi.org/10.1016/j.neucom.2018.01.077Casteleiro-Roca, J.L., Quintián, H., Calvo-Rolle, J.L, Méndez-Pérez, J. A., Perez-Castelo, F.J., Corchado, E. 2020. Lithium iron phosphate power cell fault detection system based on hybrid intelligent system. Logic Journal of the IGPL, 28(1), 71-82. https://doi.org/10.1093/jigpal/jzz072Chavero-Navarrete, E., Trejo-Perea, M., Jáuregui-Correa, J. C., Carrillo- Serrano, R. V., Ronquillo-Lomeli, G., Ríos-Moreno, J. G. 2020. Hierarchical pitch control for small wind turbines based on fuzzy logic and anticipated wind speed measurement. Applied Sciences, 10(13), 4592. https://doi.org/10.3390/app10134592Chen, P., Han, D., Tan, F., Wang, J. 2020. Reinforcement-based robust variable pitch control of wind turbines. IEEE Access 8, 20493-20502. https://doi.org/10.1109/ACCESS.2020.2968853Demirdelen, T., Tekin, P., Aksu, I. O., Ekinci, F. 2019. The prediction model of characteristics for wind turbines based on meteorological properties using neural network swarm intelligence. Sustainability, 11(17), 4803. https://doi.org/10.3390/su11174803Deng, X., Yang, J., Sun, Y., Song, D., Xiang, X., Ge, X., Joo, Y. H. 2019. Sensorless effective wind speed estimation method based on unknown input disturbance observer and extreme learning machine. Energy, 186, 115790. https://doi.org/10.1016/j.energy.2019.07.120Deng, X., Yang, J., Sun, Y., Song, D., Yang, Y., Joo, Y. H. 2020. An effective wind speed estimation based extended optimal torque control for maximum wind energy capture. IEEE Access, 8, 65959-65969. https://doi.org/10.1109/ACCESS.2020.2984654Du, J., Wang, B. 2020. Pitch Control of wind turbines based on BP neural network PI. In Journal of Physics: Conference Series (Vol. 1678, No. 1, p. 012060). IOP Publishing. https://doi.org/10.1088/1742-6596/1678/1/012060El Maati, Y. A., El Bahir, L. 2020. Optimal fault tolerant control of large-scale wind turbines in the case of the pitch actuator partial faults. Complexity. https://doi.org/10.1155/2020/6210407Fernandez-Gauna, B., Fernandez-Gamiz, U., Grana, M. 2017. Variable speed wind turbine controller adaptation by reinforcement learning. Integrated Computer-Aided Engineering 24(1), 27-39. https://doi.org/10.3233/ICA-160531Fernandez-Gauna, B., Osa, J. L., Graña, M. 2018. Experiments of conditioned reinforcement learning in continuous space control tasks. Neurocomputing 271, 38-47. https://doi.org/10.1016/j.neucom.2016.08.155Guo, C., Wang, D. 2019. Frequency regulation and coordinated control for complex wind power systems. Complexity, 2019. https://doi.org/10.1155/2019/8525397Hosseini, E., Aghadavoodi, E., Ramírez, L. M. F. 2020. Improving response of wind turbines by pitch angle controller based on gain-scheduled recurrent ANFIS type 2 with passive reinforcement learning. Renewable Energy, 157, 897-910. https://doi.org/10.1016/j.renene.2020.05.060IRENA. 2019. Future of wind: Deployment, investment, technology, grid integration and socio-economic aspects (A Global Energy Transformation paper), International Renewable Energy Agency, Abu Dhabi. https://www.irena.org/-/media/Files/IRENA/Agency/Publication/2019/Oct/IRENA_Future_of_wind_2019.pdfJeon, T., Paek, I. 2021. Design and verification of the LQR controller based on fuzzy logic for large wind turbine. Energies, 14(1), 230. https://doi.org/10.3390/en14010230Jie, W., Jingchun, C., Lin, Y., Wenliang, W., Jian, D. 2020. Pitch control of wind turbine based on deep neural network. In IOP Conference Series: Earth and Environmental Science (Vol. 619, No. 1, p. 012034). IOP Publishing https://doi.org/10.1088/1755-1315/619/1/012034Jove, E., Casteleiro‐Roca, J. L., Quintián, H., Méndez‐Pérez, J. A., Calvo‐Rolle, J. L. 2019. A fault detection system based on unsupervised techniques for industrial control loops. Expert Systems, 36(4), e12395. https://doi.org/10.1111/exsy.12395Jove, E., Casteleiro-Roca, J., Quintián, H., Méndez-Pérez, J. A., Calvo-Rolle, J. L. 2020. Detección de anomalías basada en técnicas inteligentes de una planta de obtención de material bicomponente empleado en la fabricación de palas de aerogenerador. Revista Iberoamericana de Automática e Informática industrial, 17(1), 84-93. https://doi.org/10.4995/riai.2019.11055Li, M., Wang, S. 2019. Dynamic fault monitoring of pitch system in wind turbines using selective ensemble small-world neural networks. Energies, 12(17), 3256. https://doi.org/10.3390/en12173256Mikati, M., Santos, M., Armenta, C. 2013. Electric grid dependence on the configuration of a small-scale wind and solar power hybrid system. Renewable energy, 57, 587-593. https://doi.org/10.1016/j.renene.2013.02.018Moodi, H., Bustan, D. 2019. Wind turbine control using TS systems with nonlinear consequent parts. Energy, 172, 922-931. https://doi.org/10.1016/j.energy.2019.01.133Naciones Unidas. 2021. https://sdgs.un.org/2030agenda. Accedido por última vez en 15/08/2021Ngo, Q. V., Chai, Y., Nguyen, T. T. 2020. The fuzzy-PID based-pitch angle controller for small-scale wind turbine. International Journal of Power Electronics and Drive Systems, 11(1), 135. https://doi.org/10.11591/ijpeds.v11.i1.pp135-142Our World in Data. 2021. https://ourworldindata.org/renewable-energy. Accedido por última vez en 15/08/2021.Phan, B. C., Lai, Y. C. 2019. Control strategy of a hybrid renewable energy system based on reinforcement learning approach for an isolated microgrid. Applied Sciences, 9(19), 4001. https://doi.org/10.3390/app9194001Ren, H., Hou, B., Zhou, G., Shen, L., Wei, C., Li, Q. 2020. Variable pitch active disturbance rejection control of wind turbines based on BP neural network PID. IEEE Access, 8, 71782-71797. https://doi.org/10.1109/ACCESS.2020.2987912Rubio, P. M., Quijano, J. F., López, P. Z., Lozano, J. J. F., Cerezo, A. G., Casanova, J. O. 2019. Control inteligente para mejorar el rendimiento de una plataforma semisumergible híbrida con aerogeneradores y convertidores de oleaje: sistema de control borroso para la turbina. Revista Iberoamericana de Automática e Informática industrial, 16(4), 480-491. https://doi.org/10.4995/riai.2019.10972Saénz-Aguirre, A., Zulueta, E., Fernández-Gamiz, U., Lozano, J., Lopez-Guede, J. M. 2019. Artificial neural network based reinforcement learning for wind turbine yaw control. Energies 12(3), 436. https://doi.org/10.3390/en12030436Saénz‐Aguirre, A., Zulueta, E., Fernandez‐Gamiz, U., Ulazia, A., Teso‐Fz‐Betono, D. 2020. Performance enhancement of the artificial neural network-based reinforcement learning for wind turbine yaw control. Wind Energy 23(3), 676-690. https://doi.org/10.1002/we.2451Santos, M. 2011. Un enfoque aplicado del control inteligente. Revista Iberoamericana de Automática e Informática Industrial RIAI 8(4), 283-296. https://doi.org/10.1016/j.riai.2011.09.016Sedighizadeh, M., Rezazadeh, A. 2008. Adaptive PID controller based on reinforcement learning for wind turbine control. In: Proc. World Academy of Science, Engineering and Technology 27, 257-262.Sierra, J. E., Santos, M. 2018. Modelling engineering systems using analytical and neural techniques: Hybridization. Neurocomputing, 271, 70-83. https://doi.org/10.1016/j.neucom.2016.11.099Sierra-García, J. E., Santos, M. 2020a. Performance analysis of a wind turbine pitch neurocontroller with unsupervised learning. Complexity, 2020. https://doi.org/10.1155/2020/4681767Sierra-García, J. E., Santos, M. 2020b. Exploring reward strategies for wind turbine pitch control by reinforcement learning. Applied Sciences, 10(21), 7462. https://doi.org/10.3390/app10217462Sierra-García, J. E., Santos, M. 2021a. Improving wind turbine pitch control by effective wind neuro-estimators. IEEE Access, 9, 10413-10425. https://doi.org/10.1109/ACCESS.2021.3051063Sierra-García, J. E., Santos, M. 2021b. Lookup table and neural network hybrid strategy for wind turbine pitch control. Sustainability, 13(6), 3235. https://doi.org/10.3390/su13063235Sierra-Garcia, J. E., Santos, M. 2021c. Deep learning and fuzzy logic to implement a hybrid wind turbine pitch control. Neural Computing and Applications, 1-15. https://doi.org/10.1007/s00521-021-06323-wSutton, R. S., Barto, A. G. 2015. Reinforcement learning an introduction-Second edition, in progress.Tomás-Rodríguez, M., Santos, M. 2019. Modelado y control de turbinas eólicas marinas flotantes. Revista Iberoamericana de Automática e Informática Industrial, 16(4), 381-390. https://doi.org/10.4995/riai.2019.11648Tomin, N., Kurbatsky, V., Guliyev, H. 2019. Intelligent control of a wind turbine based on reinforcement learning. In 16th Conf. on Electrical Machines, Drives and Power Systems ELMA, 1-6. IEEE. https://doi.org/10.1109/ELMA.2019.8771645Vives, J., Quiles, E., García, E. 2020. AI techniques applied to diagnosis of vibrations failures in wind turbines. IEEE Latin America Transactions, 18(08), 1478-1486. https://doi.org/10.1109/TLA.2020.9111685Zhao, H., Zhao, J., Qiu, J., Liang, G., Dong, Z. Y. 2020. Cooperative wind farm control with deep reinforcement learning and knowledge assisted learning. IEEE Transactions on Industrial Informatics. https://doi.org/10.1109/TII.2020.297403