Rnafm

🚀 RNA-FM

A pre-trained model on non-coding RNA (ncRNA) using a masked language modeling (MLM) objective.

🚀 Quick Start

The model file depends on the multimolecule library. You can install it using pip:

pip install multimolecule

✨ Features

• Pre-trained on non-coding RNA (ncRNA) with a masked language modeling (MLM) objective.
• Can be used for masked language modeling, RNA secondary structure prediction, feature extraction, and fine - tuning for various downstream tasks.

📦 Installation

You can install the required multimolecule library using pip:

pip install multimolecule

💻 Usage Examples

Basic Usage

Masked Language Modeling

You can use this model directly with a pipeline for masked language modeling:

>>> import multimolecule  # you must import multimolecule to register models
>>> from transformers import pipeline
>>> unmasker = pipeline("fill-mask", model="multimolecule/rnafm")
>>> unmasker("gguc<mask>cucugguuagaccagaucugagccu")

[{'score': 0.2752501964569092,
  'token': 21,
  'token_str': '.',
  'sequence': 'G G U C. C U C U G G U U A G A C C A G A U C U G A G C C U'},
 {'score': 0.22108642756938934,
  'token': 23,
  'token_str': '*',
  'sequence': 'G G U C * C U C U G G U U A G A C C A G A U C U G A G C C U'},
 {'score': 0.18201279640197754,
  'token': 25,
  'token_str': 'I',
  'sequence': 'G G U C I C U C U G G U U A G A C C A G A U C U G A G C C U'},
 {'score': 0.10875876247882843,
  'token': 9,
  'token_str': 'U',
  'sequence': 'G G U C U C U C U G G U U A G A C C A G A U C U G A G C C U'},
 {'score': 0.08898332715034485,
  'token': 6,
  'token_str': 'A',
  'sequence': 'G G U C A C U C U G G U U A G A C C A G A U C U G A G C C U'}]

RNA Secondary Structure Prediction

You can use this model directly with a pipeline for secondary structure prediction:

>>> import multimolecule  # you must import multimolecule to register models
>>> from transformers import pipeline
>>> predictor = pipeline("rna-secondary-structure", model="multimolecule/rnafm")
>>> predictor("ggucuc")

{'sequence': 'G G U C U C',
 'secondary_structure': '([(].)',
 'contact_map': [[0.5174561142921448, 0.49224206805229187, 0.5099285244941711, 0.4873944818973541, 0.4782561957836151, 0.49956777691841125],
  [0.49224206805229187, 0.5018295645713806, 0.4795883297920227, 0.5186688303947449, 0.4944656491279602, 0.4980457127094269],
  [0.5099285244941711, 0.4795883297920227, 0.5018683671951294, 0.4938940703868866, 0.4913065433502197, 0.5082612633705139],
  [0.4873944818973541, 0.5186688303947449, 0.4938940703868866, 0.4948427081108093, 0.479320764541626, 0.5107203722000122],
  [0.4782561957836151, 0.4944656491279602, 0.4913065433502197, 0.479320764541626, 0.5626780986785889, 0.47851765155792236],
  [0.49956777691841125, 0.4980457127094269, 0.5082612633705139, 0.5107203722000122, 0.47851765155792236, 0.48973602056503296]]}

Advanced Usage

Extract Features

Here is how to use this model to get the features of a given sequence in PyTorch:

from multimolecule import RnaTokenizer, RnaFmModel


tokenizer = RnaTokenizer.from_pretrained("multimolecule/rnafm")
model = RnaFmModel.from_pretrained("multimolecule/rnafm")

text = "UAGCUUAUCAGACUGAUGUUG"
input = tokenizer(text, return_tensors="pt")

output = model(**input)

Sequence Classification / Regression

⚠️ Important Note

This model is not fine-tuned for any specific task. You will need to fine-tune the model on a downstream task to use it for sequence classification or regression.

Here is how to use this model as backbone to fine-tune for a sequence-level task in PyTorch:

import torch
from multimolecule import RnaTokenizer, RnaFmForSequencePrediction


tokenizer = RnaTokenizer.from_pretrained("multimolecule/rnafm")
model = RnaFmForSequencePrediction.from_pretrained("multimolecule/rnafm")

text = "UAGCUUAUCAGACUGAUGUUG"
input = tokenizer(text, return_tensors="pt")
label = torch.tensor([1])

output = model(**input, labels=label)

Token Classification / Regression

⚠️ Important Note

This model is not fine-tuned for any specific task. You will need to fine-tune the model on a downstream task to use it for token classification or regression.

Here is how to use this model as backbone to fine-tune for a nucleotide-level task in PyTorch:

import torch
from multimolecule import RnaTokenizer, RnaFmForTokenPrediction


tokenizer = RnaTokenizer.from_pretrained("multimolecule/rnafm")
model = RnaFmForTokenPrediction.from_pretrained("multimolecule/rnafm")

text = "UAGCUUAUCAGACUGAUGUUG"
input = tokenizer(text, return_tensors="pt")
label = torch.randint(2, (len(text), ))

output = model(**input, labels=label)

Contact Classification / Regression

⚠️ Important Note

This model is not fine-tuned for any specific task. You will need to fine-tune the model on a downstream task to use it for contact classification or regression.

Here is how to use this model as backbone to fine-tune for a contact-level task in PyTorch:

import torch
from multimolecule import RnaTokenizer, RnaFmForContactPrediction


tokenizer = RnaTokenizer.from_pretrained("multimolecule/rnafm")
model = RnaFmForContactPrediction.from_pretrained("multimolecule/rnafm")

text = "UAGCUUAUCAGACUGAUGUUG"
input = tokenizer(text, return_tensors="pt")
label = torch.randint(2, (len(text), len(text)))

output = model(**input, labels=label)

📚 Documentation

Model Details

RNA-FM is a bert-style model pre-trained on a large corpus of non-coding RNA sequences in a self-supervised fashion. This means that the model was trained on the raw nucleotides of RNA sequences only, with an automatic process to generate inputs and labels from those texts.

Variations

• multimolecule/rnafm: The RNA-FM model pre-trained on non-coding RNA sequences.
• multimolecule/mrnafm: The RNA-FM model pre-trained on messenger RNA sequences.

Model Specification

Links

• Code: multimolecule.rnafm
• Data: RNAcentral
• Paper: Interpretable RNA Foundation Model from Unannotated Data for Highly Accurate RNA Structure and Function Predictions
• Developed by: Jiayang Chen, Zhihang Hu, Siqi Sun, Qingxiong Tan, Yixuan Wang, Qinze Yu, Licheng Zong, Liang Hong, Jin Xiao, Tao Shen, Irwin King, Yu Li
• Original Repository: ml4bio/RNA-FM

Training Details

Training Data

The RNA-FM model was pre-trained on RNAcentral. RNAcentral is a free, public resource that offers integrated access to a comprehensive and up-to-date set of non-coding RNA sequences provided by a collaborating group of Expert Databases representing a broad range of organisms and RNA types.

RNA-FM applied CD-HIT (CD-HIT-EST) with a cut-off at 100% sequence identity to remove redundancy from the RNAcentral. The final dataset contains 23.7 million non-redundant RNA sequences.

RNA-FM preprocessed all tokens by replacing "U"s with "T"s.

Note that during model conversions, "T" is replaced with "U". [RnaTokenizer][multimolecule.RnaTokenizer] will convert "T"s to "U"s for you, you may disable this behaviour by passing replace_T_with_U=False.

Training Procedure

Preprocessing

RNA-FM used masked language modeling (MLM) as the pre-training objective. The masking procedure is similar to the one used in BERT:

• 15% of the tokens are masked.
• In 80% of the cases, the masked tokens are replaced by <mask>.
• In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
• In the 10% remaining cases, the masked tokens are left as is.

PreTraining

The model was trained on 8 NVIDIA A100 GPUs with 80GiB memories.

• Learning rate: 1e-4
• Weight decay: 0.01
• Learning rate scheduler: inverse square root
• Learning rate warm-up: 10,000 steps

🔧 Technical Details

RNA-FM uses masked language modeling (MLM) for pre - training. It processes non - coding RNA sequences, pre - processes tokens by replacing "U"s with "T"s, and applies a specific masking procedure during training. The model is trained on a large and diverse dataset from RNAcentral after redundancy removal. It can be used for various RNA - related tasks either directly or after fine - tuning.

📄 License

This model is licensed under the AGPL-3.0 License.

SPDX-License-Identifier: AGPL-3.0-or-later

Citation

BibTeX:

@article{chen2022interpretable,
  title={Interpretable rna foundation model from unannotated data for highly accurate rna structure and function predictions},
  author={Chen, Jiayang and Hu, Zhihang and Sun, Siqi and Tan, Qingxiong and Wang, Yixuan and Yu, Qinze and Zong, Licheng and Hong, Liang and Xiao, Jin and King, Irwin and others},
  journal={arXiv preprint arXiv:2204.00300},
  year={2022}
}

Contact

Please use GitHub issues of MultiMolecule for any questions or comments on the model card.

Please contact the authors of the RNA-FM paper for questions or comments on the paper/model.