Climategpt 70b

🚀 ClimateGPT-70B

ClimateGPT is a series of AI models crafted to synthesize interdisciplinary research on climate change. ClimateGPT-70B, a 70-billion parameter transformer decoder model, is adapted from Llama - 2 for climate science through continuous pre - training on 4.2B tokens from curated climate documents. It's further instruction fine - tuned on a dataset of instruction - completion pairs collected by AppTek with climate scientists. ClimateGPT-7B outperforms Llama-2-70B Chat in climate - specific benchmarks. The model can be used with retrieval augmentation and cascaded machine translation to enhance knowledge, factuality, and language coverage.

🚀 Quick Start

This model is mainly for climate - specific question - answering. It can offer useful feedback for decision - makers, scientists, and journalists in climate discussions. Developers can also use it as a starting point for further fine - tuning. Note that it's not a general - purpose chatbot, though it has chat capabilities. For the full system, visit eci.io.

✨ Features

• Specialized for Climate: Adapted from Llama - 2 for climate science, it can handle climate - related questions effectively.
• Instruction Fine - Tuned: Fine - tuned on a dataset collected with climate scientists, improving its performance in climate discussions.
• Retrieval Augmentation Support: Can work with retrieval augmentation to extend knowledge and increase factuality.
• Cascaded Machine Translation: Supports cascaded machine translation to increase language coverage.

📦 Installation

No installation steps are provided in the original document, so this section is skipped.

💻 Usage Examples

No code examples are provided in the original document, so this section is skipped.

📚 Documentation

Model Details

Explore the model lineage here.

Uses

• Question - Answering: Directly used as a climate - specialized question - answering model.
• Feedback for Stakeholders: Provide useful feedback for decision - makers, scientists, and journalists in climate discussions.
• Fine - Tuning: Serve as a starting point for developers to further fine - tune.

Downstream Use

ClimateGPT - 70B can be directly used for climate - specific question - answering. It was trained to work well with retrieval augmentation and supports up to 5 references in context. When prompting, follow the ChatML format:

<|im_start|>system
{system_message}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>context
[[0]] "{reference1_title}", {reference1_year}
{reference1_text}
[[1]] "{reference2_title}", {reference2_year}
{reference2_text}
[...]<|im_end|>
<|im_start|>assistant

Training

• Llama - 2 Training Data: Refer to https://huggingface.co/meta - llama/Llama - 2 - 70b - hf.
• Continued Pre - training: 4.2B climate - specific tokens (tokenized by the Llama tokenizer) are used.
• Instruction Fine - Tuning: About 272K instruction - completion pairs (both in the climate and general domains) are used.

Evaluation

Detailed evaluation results are presented in our paper on our model card website: [eci.io/model - card](https://eci.io/model - card)

Environmental Impact

Citation

If you find ClimateGPT useful in your work, please cite it with:

@misc{thulke2024climategpt,
      title={ClimateGPT: Towards AI Synthesizing Interdisciplinary Research on Climate Change}, 
      author={David Thulke and Yingbo Gao and Petrus Pelser and Rein Brune and Rricha Jalota and Floris Fok and Michael Ramos and Ian van Wyk and Abdallah Nasir and Hayden Goldstein and Taylor Tragemann and Katie Nguyen and Ariana Fowler and Andrew Stanco and Jon Gabriel and Jordan Taylor and Dean Moro and Evgenii Tsymbalov and Juliette de Waal and Evgeny Matusov and Mudar Yaghi and Mohammad Shihadah and Hermann Ney and Christian Dugast and Jonathan Dotan and Daniel Erasmus},
      year={2024},
      eprint={2401.09646},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}

📄 License

The model is released under the ClimateGPT Community License.

⚠️ Important Note

Despite the efforts from the development team to eliminate them, as every other chat - capable LLMs, this model may generate biased, offensive or inaccurate responses.