Bertopic ArXiv

🚀 BERTopic_ArXiv

BERTopic_ArXiv is a pre - trained topic modeling model based on BERTopic. It can generate interpretable topics from large - scale ArXiv abstracts, supporting multiple topic representation methods.

🚀 Quick Start

This is a BERTopic model. BERTopic is a flexible and modular topic modeling framework that allows for the generation of easily interpretable topics from large datasets. This pre - trained model demonstrates the use of several representation models that can be used within BERTopic. It was trained on ~30000 ArXiv abstracts with various topic representation methods (bertopic.representation), including POS, KeyBERTInspired, MaximalMarginalRelevance, KeyBERT + MaximalMarginalRelevance, ChatGPT labels, and ChatGPT summaries.

An example of the default c - TF - IDF representations:

An example of labels generated by ChatGPT (gpt - 3.5 - turbo):

To generate these images, you can follow along with this tutorial:

✨ Features

• Multiple Topic Representations: Supports various topic representation methods such as POS, KeyBERTInspired, MaximalMarginalRelevance, and those integrated with ChatGPT.
• Easy to Use: Simple installation and usage process, suitable for quickly obtaining topic information.

📦 Installation

To use this model, please install BERTopic:

pip install -U bertopic
pip install -U safetensors

💻 Usage Examples

Basic Usage

from bertopic import BERTopic
topic_model = BERTopic.load("MaartenGr/BERTopic_ArXiv")

topic_model.get_topic_info()

Advanced Usage

To view all different topic representations (keywords, labels, summary, etc.) you can run the following:

>>> topic_model.get_topic(0, full=True)
{'Main': [['dialogue', 0.02704485163341523],
  ['dialog', 0.01677038224466311],
  ['response', 0.011692640237477233],
  ['responses', 0.01002788412923778],
  ['intent', 0.00990720856306287],
  ['oriented', 0.009217253131615378],
  ['slot', 0.009177118721490055],
  ['conversational', 0.009129311385144046],
  ['systems', 0.009101146153425574],
  ['conversation', 0.008845392252307181]],
 'POS': [['dialogue', 0.02704485163341523],
  ['dialog', 0.01677038224466311],
  ['response', 0.011692640237477233],
  ['responses', 0.01002788412923778],
  ['intent', 0.00990720856306287],
  ['slot', 0.009177118721490055],
  ['conversational', 0.009129311385144046],
  ['systems', 0.009101146153425574],
  ['conversation', 0.008845392252307181],
  ['user', 0.008753551043296965]],
 'KeyBERTInspired': [['task oriented dialogue', 0.6559894680976868],
  ['dialogue systems', 0.6249060034751892],
  ['oriented dialogue', 0.5788208246231079],
  ['dialog systems', 0.530449628829956],
  ['dialogue state', 0.5167528390884399],
  ['response generation', 0.5143576860427856],
  ['spoken language understanding', 0.46739083528518677],
  ['oriented dialog', 0.4600704610347748],
  ['dialog', 0.4534587264060974],
  ['dialogues', 0.44082391262054443]],
 'MMR': [['dialogue', 0.02704485163341523],
  ['dialog', 0.01677038224466311],
  ['response', 0.011692640237477233],
  ['responses', 0.01002788412923778],
  ['intent', 0.00990720856306287],
  ['oriented', 0.009217253131615378],
  ['slot', 0.009177118721490055],
  ['conversational', 0.009129311385144046],
  ['systems', 0.009101146153425574],
  ['conversation', 0.008845392252307181]],
 'KeyBERT + MMR': [['task oriented dialogue', 0.6559894680976868],
  ['dialogue systems', 0.6249060034751892],
  ['oriented dialogue', 0.5788208246231079],
  ['dialog systems', 0.530449628829956],
  ['dialogue state', 0.5167528390884399],
  ['response generation', 0.5143576860427856],
  ['spoken language understanding', 0.46739083528518677],
  ['oriented dialog', 0.4600704610347748],
  ['dialog', 0.4534587264060974],
  ['dialogues', 0.44082391262054443]],
 'OpenAI_Label': [['Challenges and Approaches in Developing Task-oriented Dialogue Systems',
   1]],
 'OpenAI_Summary': [['Task-oriented dialogue systems and their components, such as dialogue policy, natural language understanding, dialogue state tracking, response generation, and end-to-end training using neural networks. These components are crucial in assisting users to complete various activities such as booking tickets and restaurant reservations through spoken language understanding dialogue. The challenge lies in tracking dialogue states of multiple domains and obtaining annotations for training. Effective SLU is achieved by utilizing context from the prior dialogue history.',
   1]]}

📚 Documentation

Topic overview

• Number of topics: 107
• Number of training documents: 33189

🔧 Technical Details

Training Procedure

The model was trained as follows:

from cuml.manifold import UMAP
from cuml.cluster import HDBSCAN
from bertopic import BERTopic
from sklearn.feature_extraction.text import CountVectorizer
from bertopic.representation import PartOfSpeech, KeyBERTInspired, MaximalMarginalRelevance, OpenAI

# Prepare sub-models
embedding_model = SentenceTransformer('sentence-transformers/all-mpnet-base-v2')
umap_model = UMAP(n_components=5, n_neighbors=50, random_state=42, metric="cosine", verbose=True)
hdbscan_model = HDBSCAN(min_samples=20, gen_min_span_tree=True, prediction_data=False, min_cluster_size=20, verbose=True)
vectorizer_model = CountVectorizer(stop_words="english", ngram_range=(1, 3), min_df=5)

# Summarization with ChatGPT
summarization_prompt = """
I have a topic that is described by the following keywords: [KEYWORDS]
In this topic, the following documents are a small but representative subset of all documents in the topic:
[DOCUMENTS]

Based on the information above, please give a description of this topic in the following format:
topic: <description>
"""
summarization_model = OpenAI(model="gpt-3.5-turbo", chat=True, prompt=summarization_prompt, nr_docs=5, exponential_backoff=True, diversity=0.1)

# Representation models
representation_models = {
    "POS": PartOfSpeech("en_core_web_lg"),
    "KeyBERTInspired": KeyBERTInspired(),
    "MMR": MaximalMarginalRelevance(diversity=0.3),
    "KeyBERT + MMR": [KeyBERTInspired(), MaximalMarginalRelevance(diversity=0.3)],
    "OpenAI_Label": OpenAI(model="gpt-3.5-turbo", exponential_backoff=True, chat=True, diversity=0.1),
    "OpenAI_Summary": [KeyBERTInspired(), summarization_model],
}

# Fit BERTopic
topic_model= BERTopic(
        embedding_model=embedding_model,
        umap_model=umap_model,
        hdbscan_model=hdbscan_model,
        vectorizer_model=vectorizer_model,
        representation_model=representation_models,
        verbose=True
).fit(docs)

Training hyperparameters

Framework versions