DUO Open-source Pretrained Model - Used for natural language processing, capable of completing masked language modeling for free.

Duo

Developed by s-sahoo

DUO is a pretrained model based on the Transformers library, focusing on masked language modeling tasks and suitable for the field of natural language processing.

Large Language Model

Transformers

EnglishOpen Source License:Apache-2.0 #Masked Language Modeling #Diffusion Duality Technique #Long Text Processing

Downloads 212

Release Time : 2/18/2025

Model Overview

The DUO model is a pretrained model for masked language modeling, with a parameter scale similar to that of GPT2-medium, and it has been trained on the OpenWebText corpus.

Model Features

Large Context Length

Supports a context length of 1024, suitable for long text processing tasks.

Efficient Training

Trained for 1 million steps on the OpenWebText corpus to optimize model performance.

Similar Scale to GPT2-medium

The parameter scale is similar to that of GPT2-medium, approximately 130 million non-embedded parameters.

Model Capabilities

Masked Language Modeling

Text Generation

Natural Language Understanding

Use Cases

Natural Language Processing

Text Completion

Use the masked language modeling ability to complete the missing text parts.

Language Model Fine-tuning

As a base model, it can be used for fine-tuning on specific tasks.

🚀 DUO

DUO is a pre - trained model for masked language modeling. It offers high - quality performance and can be easily integrated into various NLP tasks.

🚀 Quick Start

To use the pre - trained model for masked language modeling, use the following snippet:

from transformers import AutoModelForMaskedLM, AutoTokenizer

# See the `MDLM` collection page on the hub for list of available models.
tokenizer = transformers.AutoTokenizer.from_pretrained('gpt2')
model = AutoModelForMaskedLM.from_pretrained('s - sahoo/duo')

For a hands - on example, check out this [Colab notebook](https://colab.research.google.com/drive/1Sf7R - dqdR6gq - H8nyZ9E3ZkyvqMTqcwq?usp=sharing). For more information and implementation details, visit our github repository: [DUO](https://github.com/s - sahoo/duo)

✨ Features

Context Length: The model has a context length of 1024.
Model Size: It is similar in size to GPT2 - medium with approximately 130 million non - embedding parameters.
Training Data: The model was trained for 1M steps on the OpenWebText corpus from Skylion007/openwebtext.
Evaluation Metric: Perplexity is used as the evaluation metric.

📚 Documentation

For more details, please see our paper: The Diffusion Duality.

📄 License

This project is licensed under the [Apache - 2.0](apache - 2.0) license.

📦 Model Information

Property	Details
Library Name	transformers
Model Type	Pre - trained model for masked language modeling
Training Data	Skylion007/openwebtext
Metrics	perplexity

📄 Citation

Please cite our work using the bibtex below:

BibTeX:

@inproceedings{
sahoo2025the,
title={The Diffusion Duality},
author={Subham Sekhar Sahoo and Justin Deschenaux and Aaron Gokaslan and Guanghan Wang and Justin T Chiu and Volodymyr Kuleshov},
booktitle={ICLR 2025 Workshop on Deep Generative Model in Machine Learning: Theory, Principle and Efficacy},
year={2025},
url={https://openreview.net/forum?id=CB0Ub2yXjC}
}

📞 Model Card Contact

Subham Sekhar Sahoo (ssahoo@cs.cornell.edu)

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご