twhin-bert-base Open-source Multilingual Tweet Model - Optimizing Social Language Processing with 7 Billion Tweets

Twhin Bert Base

Developed by Twitter

TwHIN-BERT is a multilingual tweet language model trained on 7 billion tweets, optimized with text self-supervised learning and social interaction data.

Large Language Model

Transformers

Supports Multiple LanguagesOpen Source License:Apache-2.0 #Multilingual Tweet Understanding #Social Behavior Prediction #Heterogeneous Information Network

Downloads 3,789

Release Time : 10/18/2022

Model Overview

TwHIN-BERT is a novel multilingual tweet language model trained using rich social interaction data from Twitter's Heterogeneous Information Network (TwHIN) combined with text self-supervised learning, suitable for various NLP and recommendation tasks.

Model Features

Multilingual Support

Supports over 100 languages, particularly suitable for processing multilingual tweet content.

Social-Enhanced Training

Optimized with social interaction data from Twitter's Heterogeneous Information Network (TwHIN), enhancing performance in social recommendation tasks.

Dual Training Objectives

Trained using both text self-supervised learning (e.g., masked language modeling) and social objective functions.

Model Capabilities

Multilingual Text Understanding

Social Recommendation

Text Classification

User Behavior Prediction

Use Cases

Natural Language Processing

Multilingual Text Classification

Classify tweet content with support for multiple languages.

Outperforms peer models in semantic understanding tasks

Social Recommendation

User-Tweet Interaction Prediction

Predict user interactions with tweets.

Excels in social recommendation tasks

🚀 TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations

TwHIN-BERT is a pre - trained language model designed for multilingual tweet representations. It's trained on a large - scale dataset of tweets, enabling it to handle various NLP and recommendation tasks effectively, outperforming similar models in both semantic understanding and social recommendation.

🚀 Quick Start

This repo contains models, code and pointers to datasets from our paper: TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations. [PDF] [HuggingFace Models]

✨ Features

Multilingual Support: TwHIN-BERT is trained on 7 billion Tweets from over 100 distinct languages, including en, ja, pt, es, ko, etc.
Social - Enriched Training: It's trained not only with text - based self - supervision (e.g., MLM) but also with a social objective based on the rich social engagements within a Twitter Heterogeneous Information Network (TwHIN).
Versatile Usage: Can be used as a drop - in replacement for BERT in a variety of NLP and recommendation tasks, excelling in both semantic understanding and social recommendation tasks.

📦 Installation

We initially release two pretrained TwHIN-BERT models (base and large) that are compatible with the HuggingFace BERT models.

Property	Details
Model Type	TwHIN - BERT - base, TwHIN - BERT - large
Size	280M parameters (base), 550M parameters (large)
Download Link (🤗 HuggingFace)	[Twitter/TwHIN - BERT - base](https://huggingface.co/Twitter/twhin - bert - base), [Twitter/TwHIN - BERT - large](https://huggingface.co/Twitter/twhin - bert - large)

💻 Usage Examples

Basic Usage

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained('Twitter/twhin-bert-base')
model = AutoModel.from_pretrained('Twitter/twhin-bert-base')
inputs = tokenizer("I'm using TwHIN-BERT! #TwHIN-BERT #NLP", return_tensors="pt")
outputs = model(**inputs)

📄 License

This project is licensed under the "apache - 2.0" license.

📚 Documentation

Citation

If you use TwHIN-BERT or our datasets in your work, please cite the following:

@article{zhang2022twhin,
  title={TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations},
  author={Zhang, Xinyang and Malkov, Yury and Florez, Omar and Park, Serim and McWilliams, Brian and Han, Jiawei and El-Kishky, Ahmed},
  journal={arXiv preprint arXiv:2209.07562},
  year={2022}
}

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご