Timer Base 84m

🚀 Time-Series Transformer (Timer)

A large time-series model introduced in a research paper and enhanced through further work, serving as a lightweight generative Transformer for zero-shot point forecasting.

The Time-Series Transformer (Timer) is a large time-series model introduced in this paper and enhanced with our further work. This version is pre-trained on 260B time points with 84M parameters, making it a lightweight generative Transformer for zero-shot point forecasting.

We evaluate the model on the following benchmark: TSLib Dataset. For more information, please see the Github Repo.

There's indeed room for improvement in this small model. We are actively working around it and are glad to see constructive suggestions and noteworthy cases :)

🚀 Quick Start

Installation

pip install transformers==4.40.1 # Use this version and Python 3.10 for stable compatibility

Usage Example

import torch
from transformers import AutoModelForCausalLM

# load pretrain model
model = AutoModelForCausalLM.from_pretrained('thuml/timer-base-84m', trust_remote_code=True)

# prepare input
batch_size, lookback_length = 1, 2880
seqs = torch.randn(batch_size, lookback_length)

# generate forecast
prediction_length = 96
output = model.generate(seqs, max_new_tokens=prediction_length)

print(output.shape)

A notebook example is also provided here. Try it out!

✨ Features

• Lightweight Design: With only 84M parameters, it's a lightweight generative Transformer for zero-shot point forecasting.
• Large Pre-training Scale: Pre-trained on 260B time points, enabling better generalization.
• High Context Length: Supports a context length of up to 2880, suitable for long-term time series forecasting.

📦 Installation

pip install transformers==4.40.1 # Use this version and Python 3.10 for stable compatibility

💻 Usage Examples

Basic Usage

import torch
from transformers import AutoModelForCausalLM

# load pretrain model
model = AutoModelForCausalLM.from_pretrained('thuml/timer-base-84m', trust_remote_code=True)

# prepare input
batch_size, lookback_length = 1, 2880
seqs = torch.randn(batch_size, lookback_length)

# generate forecast
prediction_length = 96
output = model.generate(seqs, max_new_tokens=prediction_length)

print(output.shape)

📚 Documentation

Specification

Datasets

• thuml/UTSD
• Salesforce/lotsa_data

Metrics

• Mean Absolute Error (MAE)
• Mean Squared Error (MSE)

Tags

• Time Series
• Forecasting
• Foundation Models
• Pretrained Models
• Time Series Foundation Models
• Large Time Series Models

🔧 Technical Details

The Time-Series Transformer (Timer) is a causal Transformer (decoder-only) architecture. It is pre-trained on a large scale of 260B time points, enabling it to capture complex patterns in time series data. The model supports a context length of up to 2880, making it suitable for long-term time series forecasting.

📄 License

This model is licensed under the Apache-2.0 License.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (62022050 and U2342217), the BNRist Innovation Fund (BNR2024RC01010), and the National Engineering Research Center for Big Data Software.

The model is mostly built from the Internet public time series dataset, which comes from different research teams and providers. We sincerely thank all individuals and organizations who have contributed the data. Without their generous sharing, this model would not have existed.

Citation

@inproceedings{liutimer,
  title={Timer: Generative Pre-trained Transformers Are Large Time Series Models},
  author={Liu, Yong and Zhang, Haoran and Li, Chenyu and Huang, Xiangdong and Wang, Jianmin and Long, Mingsheng},
  booktitle={Forty-first International Conference on Machine Learning}
}

@article{liu2024timer,
  title={Timer-XL: Long-Context Transformers for Unified Time Series Forecasting},
  author={Liu, Yong and Qin, Guo and Huang, Xiangdong and Wang, Jianmin and Long, Mingsheng},
  journal={arXiv preprint arXiv:2410.04803},
  year={2024}
}

Contact

If you have any questions or want to use the code, feel free to contact:

• Yong Liu (liuyong21@mails.tsinghua.edu.cn)
• Guo Qin (qinguo24@mails.tsinghua.edu.cn)