Canine C

🚀 CANINE-c (CANINE pre-trained with autoregressive character loss)

CANINE-c is a pre - trained model on 104 languages using masked language modeling. It doesn't need an explicit tokenizer and operates at the character level, offering a unique approach for language representation.

🚀 Quick Start

How to use

Here is how to use this model:

from transformers import CanineTokenizer, CanineModel

model = CanineModel.from_pretrained('google/canine-c')
tokenizer = CanineTokenizer.from_pretrained('google/canine-c')

inputs = ["Life is like a box of chocolates.", "You never know what you gonna get."]
encoding = tokenizer(inputs, padding="longest", truncation=True, return_tensors="pt")

outputs = model(**encoding) # forward pass
pooled_output = outputs.pooler_output
sequence_output = outputs.last_hidden_state

✨ Features

• Tokenization - Free: CANINE doesn't require an explicit tokenizer (such as WordPiece or SentencePiece) like other models. It directly operates at a character level, turning each character into its Unicode code point.
• Multilingual: Pretrained on a large corpus of multilingual data in 104 languages, enabling it to learn an inner representation of multiple languages.
• Self - Supervised Learning: Pretrained with masked language modeling (MLM) and next sentence prediction (NSP) objectives, which helps the model learn useful features for downstream tasks.

📚 Documentation

Model description

CANINE is a transformers model pretrained on a large corpus of multilingual data in a self - supervised fashion, similar to BERT. It was pretrained with two objectives:

• Masked language modeling (MLM): One randomly masks part of the inputs, which the model needs to predict. This model (CANINE - c) is trained with an autoregressive character loss. One masks several character spans within each sequence, which the model then autoregressively predicts.
• Next sentence prediction (NSP): The model concatenates two sentences as inputs during pretraining. Sometimes they correspond to sentences that were next to each other in the original text, sometimes not. The model then has to predict if the two sentences were following each other or not.

This way, the model learns an inner representation of multiple languages that can then be used to extract features useful for downstream tasks.

Intended uses & limitations

You can use the raw model for either masked language modeling or next sentence prediction, but it's mostly intended to be fine - tuned on a downstream task. See the model hub to look for fine - tuned versions on a task that interests you.

Note that this model is primarily aimed at being fine - tuned on tasks that use the whole sentence (potentially masked) to make decisions, such as sequence classification, token classification or question answering. For tasks such as text generation you should look at models like GPT2.

🔧 Technical Details

Training data

The CANINE model was pretrained on the multilingual Wikipedia data of mBERT, which includes 104 languages.

BibTeX entry and citation info

@article{DBLP:journals/corr/abs-2103-06874,
  author    = {Jonathan H. Clark and
               Dan Garrette and
               Iulia Turc and
               John Wieting},
  title     = {{CANINE:} Pre-training an Efficient Tokenization-Free Encoder for
               Language Representation},
  journal   = {CoRR},
  volume    = {abs/2103.06874},
  year      = {2021},
  url       = {https://arxiv.org/abs/2103.06874},
  archivePrefix = {arXiv},
  eprint    = {2103.06874},
  timestamp = {Tue, 16 Mar 2021 11:26:59 +0100},
  biburl    = {https://dblp.org/rec/journals/corr/abs-2103-06874.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}

📄 License

The license for this model is apache - 2.0.

Information Table