Modern Neural Networks for Time-Series Forecasting (Transformers vs. LSTMs)

Transformer network was first introduced in 2017 in the paper: Attention is all you need. They solve sequence-to-sequence tasks and are an improvement over Long Short Term Memory (LSTM) because they can handle a long range of dependencies. All of the previous architectures executed sequentially and did not use the GPU efficiently but transformers solved that problem with the multi-headed attention architecture. In this talk we will compare the, 1. Architectural differences between LSTM and Transformers. 2. Performance of LSTM vs. Transformer for a time series forecasting task based on the following criteria: a. Accuracy of prediction b. Complexity of the architecture c. Time to trainhttps://pdxscholar.library.pdx.edu/systems_science_seminar_series/1131/thumbnail.jp