[ad_1]
Uncover the structure of iTransformer and apply the mannequin in a small experiment utilizing Python.
Picture by David Clode on Unsplash
The sector of forecasting has seen plenty of exercise within the realm of basis fashions, with fashions like Lag-LLaMA, Time-LLM, Chronos and Moirai being proposed because the starting of 2024.
Nevertheless, their efficiency has been a bit underwhelming (for reproducible benchmarks, see right here), and I consider that data-specific fashions are nonetheless the optimum answer for the time being.
To that finish, the Transformer structure has been utilized in lots of kinds for time collection forecasting, with PatchTST attaining state-of-the-art efficiency for long-horizon forecasting.
Difficult PatchTST comes the iTransformer mannequin, proposed in March 2024 within the paper iTransformer: Inverted Transformers Are Efficient for Time Sequence Forecasting.
On this article, we uncover the strikingly easy idea behind iTransformer and discover its structure. Then, we apply the mannequin in a small experiment and evaluate its efficiency to TSMixer, N-HiTS and PatchTST.
For extra particulars, be sure that to learn the unique paper.
Let’s get began!
The concept behind iTransformer comes from the belief that the vanilla Transformer mannequin makes use of temporal tokens.
Which means that the mannequin appears to be like in any respect options at a single time step. Thus, it’s difficult for the mannequin to be taught temporal dependencies when one time step at a time.
An answer to that drawback is patching, which was proposed with the PatchTST mannequin. With patching, we merely group time factors collectively earlier than tokenizing and embedding them, as proven beneath.
Visualizing patching. Right here, we now have a sequence of 15 timesteps, with a patch size of 5 and a stride of 5 as effectively, leading to three patches. Picture by the writer.
In iTransformer, we push patching to the acute by merely making use of the mannequin on the inverted dimensions.
[ad_2]
Supply hyperlink
