BERT-for-Patents-64D Open-Source Patent Text Processing Model - Optimizing Storage for Efficient Handling of Patent Information

Bert For Patents 64d

Developed by prithivida

Patent text processing model based on BERT-LARGE architecture, optimizing storage efficiency through PCA-compressed embedding dimensions

Large Language Model

Transformers

EnglishOpen Source License:Apache-2.0 #Patent Text Embedding #PCA Dimensionality Reduction Optimization #Multilingual Patent Analysis

Downloads 174

Release Time : 3/31/2022

Model Overview

BERT model specifically trained for patent literature, supporting semantic understanding and masked prediction tasks for patent texts, reducing embedding storage requirements via dimensionality compression

Model Features

Patent Domain Optimization

Trained on 100M+ global patent data, specially optimized for patent texts

Storage Efficiency Optimization

Compresses 1024-dimensional embeddings to 64 dimensions via PCA, reducing storage space by 93.75%

Technical Term Understanding

Excels at processing specialized terminology and technical descriptions in patent literature

Model Capabilities

Patent Text Semantic Understanding

Technical Term Prediction

Patent Text Masked Completion

Use Cases

Patent Analysis

Patent Technical Element Extraction

Automatically identifies key technical component descriptions in patent literature

Improves patent search and analysis efficiency

Patent Similarity Calculation

Compares patent technical similarity via embedding vectors

Supports patent infringement detection and technology evolution analysis

Intellectual Property Management

Patent Classification

Automatically categorizes patent literature by technical field

Optimizes patent portfolio management

🚀 BERT for Patents with Reduced Embeddings

This project is based on BERT for Patents and reduces the embedding dimension to save storage space while maintaining comparable performance.

🚀 Quick Start

This README provides details about a model based on anferico/bert - for - patents. It explains the motivation behind reducing the embedding dimension and provides information about the BERT for Patents model.

✨ Features

Embedding Dimension Reduction: Reduces the embedding dimension from 1024 to 64 using Principle Component Analysis (PCA), saving storage space for embedding storage.
Comparable Performance: Despite the dimension reduction, the model still offers comparable performance.

📚 Documentation

Motivation

This model is based on anferico/bert - for - patents - a BERT_LARGE model (See next section for details below). By default, the pre - trained model's output embeddings have a size of 768 (base - models) or 1024 (large - models). However, when you store millions of embeddings, this can require quite a lot of memory/storage. So, we have reduced the embedding dimension to 64, i.e., 1/16th of 1024 using Principle Component Analysis (PCA), and it still gives a comparable performance. Yes! PCA gives better performance than NMF. Note: This process neither improves the runtime nor the memory requirement for running the model. It only reduces the needed space to store embeddings, for example, for semantic search using vector databases.

BERT for Patents

BERT for Patents is a model trained by Google on 100M+ patents (not just US patents).

If you want to learn more about the model, check out the [blog post](https://cloud.google.com/blog/products/ai - machine - learning/how - ai - improves - patent - analysis), white paper and [GitHub page](https://github.com/google/patents - public - data/blob/master/models/BERT%20for%20Patents.md) containing the original TensorFlow checkpoint.

Projects using this model (or variants of it)

[Patents4IPPC](https://github.com/ec - jrc/Patents4IPPC) (carried out by [Pi School](https://picampus - school.com/) and commissioned by the Joint Research Centre (JRC) of the European Commission)

📄 License

This project is licensed under the apache - 2.0 license.

Property	Details
Model Type	BERT - based model with reduced embeddings
Training Data	100M+ patents
License	apache - 2.0
Metrics	perplexity
Pipeline Tag	"fill - mask"
Mask Token	"[MASK]"

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご