TOD-XLMR Open-Source Multilingual Task-Oriented Dialogue Model

TOD XLMR

Developed by umanlp

TOD-XLMR is a multilingual task-oriented dialogue model developed based on XLM-RoBERTa, employing a dual-objective joint training strategy to enhance dialogue understanding capabilities

Large Language Model

Transformers

OtherOpen Source License:MIT #Multilingual Dialogue Understanding #Task-Oriented Dialogue #Contrastive Learning Pretraining

Downloads 54

Release Time : 4/21/2022

Model Overview

This model specializes in optimizing semantic understanding for multilingual task-oriented dialogue scenarios through joint training of masked language modeling and response contrastive loss, supporting multilingual dialogue processing

Model Features

Multilingual Dialogue Optimization

Optimized specifically for multilingual task-oriented dialogue scenarios based on the XLM-RoBERTa architecture

Dual-Objective Joint Training

Joint training strategy combining Masked Language Modeling (MLM) and Response Contrastive Loss (RCL)

Dialogue Structure Understanding

Enhances the model's ability to capture temporal relationships in dialogues through Response Contrastive Loss

Model Capabilities

Multilingual Text Understanding

Dialogue Semantic Encoding

Task-Oriented Dialogue Processing

Use Cases

Intelligent Customer Service Systems

Multilingual Customer Service Dialogue Understanding

Used to understand customer inquiry intent in multilingual environments

Improves semantic understanding accuracy in multilingual dialogue systems

Dialogue System Development

Task-Oriented Dialogue Systems

Serves as the semantic understanding module for dialogue systems

Enhances the system's ability to understand user intent

🚀 TOD-XLMR

TOD-XLMR is a conversationally specialized multilingual model based on XLM-RoBERTa, pre - trained on English conversational corpora for task - oriented dialogue.

🚀 Quick Start

TOD-XLMR is a conversationally specialized multilingual version based on XLM-RoBERTa. It is pre-trained on English conversational corpora consisting of nine human-to-human multi-turn task-oriented dialog (TOD) datasets as proposed in the paper TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue by Wu et al. and first released in this repository.

The model is jointly trained with two objectives as proposed in TOD-BERT, including masked language modeling (MLM) and response contrastive loss (RCL). Masked language modeling is a common pretraining strategy utilized for BERT-based architectures, where a random sample of tokens in the input sequence is replaced with the special token [MASK] for predicting the original masked tokens. To further encourage the model to capture dialogic structure (i.e., dialog sequential order), response contrastive loss is implemented by using in-batch negative training with contrastive learning.

💻 Usage Examples

Basic Usage

Here is how to use this model to get the features of a given text in PyTorch:

from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("umanlp/TOD-XLMR")
model = AutoModelForMaskedLM.from_pretrained("umanlp/TOD-XLMR")

# prepare input
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')

# forward pass
output = model(**encoded_input)

Advanced Usage

Or you can also use AutoModel to load the pretrained model and further apply to downstream tasks:

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("umanlp/TOD-XLMR")
model = AutoModel("umanlp/TOD-XLMR")

# prepare input
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')

# forward pass
output = model(**encoded_input)

📄 License

The license of this project is MIT.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご