MatTPUSciBERT Open-Source Language Model - Designed Specifically to Enhance Text Comprehension Ability in Materials Science

Mattpuscibert

Developed by lfoppiano

A language model pretrained on 700,000 full-text papers in materials science based on the SciBERT framework, specifically designed to enhance text comprehension in materials science

Large Language Model

Transformers

#Materials Science NLP #TPU-accelerated Pretraining #Superconductor Identification

Downloads 161

Release Time : 9/21/2022

Model Overview

MatTPUSciBERT is a BERT model optimized for materials science, significantly improving named entity recognition and physical quantity extraction capabilities in materials science literature through domain-specific pretraining and vocabulary expansion

Model Features

Materials Science Domain Optimization

Pretrained on 700,000 full-text materials science papers, significantly enhancing domain text comprehension

Expanded Domain Vocabulary

Extended the original SciBERT vocabulary with 100 materials science-specific terms extracted using KeyBERT

TPU-efficient Training

Two-phase training (800k steps + 100k steps) conducted on Google Cloud TPU to optimize training efficiency

Multi-task Validation

Model performance validated on two typical tasks: superconductor named entity recognition and physical quantity extraction

Model Capabilities

Materials science text comprehension

Superconductor named entity recognition

Physical quantity extraction

Scientific literature information extraction

Use Cases

Materials Science Research

Superconductor Material Discovery

Automatically identify new superconductor materials and their properties from scientific literature

Achieved F1-score of 83.61%, outperforming comparable models

Material Property Quantitative Analysis

Automatically extract physical quantity data of materials reported in literature

Achieved F1-score of 87.46%, comparable to baseline models

Scientific Literature Mining

Materials Database Construction

Automatically extract structured material data from large volumes of literature

🚀 Material SciBERT (TPU): Improving language understanding in materials science

This project aims to enhance language understanding in materials science through a pre - trained SciBERT - based model.

🚀 Quick Start

This is a work in progress. The project focuses on a SciBERT - based model pre - trained with materials science scientific full - text.

✨ Features

Data Collection: Collected full - text from approximately 700,000 articles provided by the National Institute for Materials Science (NIMS) TDM platform (https://dice.nims.go.jp/services/TDM - PF/en/), forming a dataset called ScienceCorpus (SciCorpus).
Vocabulary Expansion: Added 100 domain - specific unknown words extracted from SciCorpus using a keywords modeler (KeyBERT) to the SciBERT vocabulary (32k tokens).
Pre - training: Pre - trained the model MatTpuSciBERT on Google Cloud with the TPU (Tensor Processing Unit) under specific conditions:
- 800,000 steps with batch_size: 256 and max_seq_length: 512.
- 100,000 steps with batch_size: 2048 and max_seq_length: 128.
Fine - tuning and Testing: Conducted fine - tuning and testing on NER for superconductors (https://github.com/lfoppiano/grobid - superconductors) and physical quantities (https://github.com/kermitt2/grobid - quantities).

📚 Documentation

Authors

Luca Foppiano
Pedro Ortiz Suarez

Related work

BERT Implementations

BERT (the original) https://arxiv.org/abs/1810.04805
RoBERTa (Re - implementation by Facebook) https://arxiv.org/abs/1907.11692

Relevant models

SciBERT: BERT, trained from scratch on scientific articles (biology + CS) https://github.com/allenai/scibert
MatSciBERT (Gupta): RoBERTa, trained from scratch, using SciBERT vocab and weights, with ~150K papers limited to 4 MS families http://github.com/m3rg - iitd/matscibert
MaterialBERT: Not yet published
MatBERT (CEDER): BERT, trained from scratch on 2M documents in materials science (~60M paragraphs) https://github.com/lbnlp/MatBERT
BatteryBERT (Cole): BERT, a mix of training from scratch and using predefined weights https://github.com/ShuHuang/batterybert/

Results

Results were obtained via 10 - fold cross - validation, using DeLFT (https://github.com/kermitt2/delft).

NER Superconductors

Model	Precision	Recall	F1
SciBERT (baseline)	81.62%	84.23%	82.90%
MatSciBERT (Gupta)	81.45%	84.36%	82.88%
MatTPUSciBERT	82.13%	85.15%	83.61%
MatBERT (Ceder)	81.25%	83.99%	82.60%
BatteryScibert - cased	81.09%	84.14%	82.59%

NER Quantities

Model	Precision	Recall	F1
SciBERT (baseline)	88.73%	86.76%	87.73%
MatSciBERT (Gupta)	84.98%	90.12%	87.47%
MatTPUSciBERT	88.62%	86.33%	87.46%
MatBERT (Ceder)	85.08%	89.93%	87.44%
BatteryScibert - cased	85.02%	89.30%	87.11%
BatteryScibert - cased	81.09%	84.14%	82.59%

References

This work was supported by Google, through the researchers program https://cloud.google.com/edu/researchers

Acknowledgements

TBA

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご