Nomic V2 Tuned 1

🚀 Tuned Nomic V2

This is a sentence-transformers model finetuned from nomic-ai/nomic-embed-text-v2-moe on the json dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

✨ Features

• Semantic Understanding: Maps sentences and paragraphs to a 768 - dimensional dense vector space, enabling accurate semantic similarity calculations.
• Versatile Applications: Can be used for various NLP tasks such as semantic search, paraphrase mining, text classification, and clustering.
• Fine - Tuned Performance: Finetuned on a json dataset for better performance on specific tasks.

📦 Installation

First, you need to install the Sentence Transformers library:

pip install -U sentence-transformers

💻 Usage Examples

Basic Usage

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'The company "Alpha" signed a contract with engineer Petrovsky to develop new software. During the work, Petrovsky created a patent - protected invention. The contract did not contain clauses regarding intellectual property rights. Who is the patent holder of the invention, and can the company "Alpha" use this invention without paying additional compensation?',
    '<p>1. The right to obtain a patent and the exclusive right to an invention, utility model, or industrial design created during the execution of a contract for work or a contract for scientific research, experimental design, or technological work that did not directly provide for their creation belong to the contractor (executor), unless otherwise provided by the contract between them and the customer. (As amended by Federal Law <a href="102171743">dated 12.03.2014 No. 35 - FZ</a>)</p><p>In this case, the customer has the right, unless otherwise provided by the contract, to use the invention, utility model, or industrial design created in this way for the purposes for which the corresponding contract was concluded, on the terms of a simple (non - exclusive) license throughout the term of the patent without paying additional compensation for this use. When the contractor (executor) transfers the right to obtain a patent or alienates the patent itself to another person, the customer retains the right to use the invention, utility model, or industrial design on the specified terms.</p><p>2. In the case where, in accordance with the contract between the contractor (executor) and the customer, the right to obtain a patent or the exclusive right to an invention, utility model, or industrial design is transferred to the customer or a third party specified by the customer, the contractor (executor) has the right to use the created invention, utility model, or industrial design for their own needs on the terms of a free simple (non - exclusive) license throughout the term of the patent, unless otherwise provided by the contract.</p><p>3. The author of the invention, utility model, or industrial design specified in paragraph 1 of this article, who is not the patent holder, is paid compensation in accordance with paragraph 4 of Article 1370 of this Code.</p>',
    '<p>In the transfer in the manner established by Part 11 of Article 154 of Federal Law <a href="102088491">dated 22 August 2004 No. 122 - FZ</a> "On Amending the Legislative Acts of the Russian Federation and Recognizing Certain Legislative Acts of the Russian Federation as Having Lost Force in Connection with the Adoption of Federal Laws <a href="102067003">"On Amending and Supplementing the Federal Law "On the General Principles of the Organization of the Legislative (Representative) and Executive Bodies of State Power of the Subjects of the Russian Federation"</a> and <a href="102083574">"On the General Principles of the Organization of Local Self - Government in the Russian Federation"</a>, shares from federal property to the property of a subject of the Russian Federation or municipal property, from the property of a subject of the Russian Federation to federal property or municipal property, from municipal property to federal property or the property of a subject of the Russian Federation. (Added by paragraph - Federal Law <a href="102157372">dated 14.06.2012 No. 77 - FZ</a>)</p>'
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

📚 Documentation

Model Details

Model Description

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: NomicBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

🔧 Technical Details

Evaluation

Metrics

Information Retrieval

• Datasets: dim_768, dim_512, dim_256, dim_128 and dim_64
• Evaluated with InformationRetrievalEvaluator

Training Details

Training Dataset

json

• Dataset: json
• Size: 13,186 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 18 tokens mean: 61.09 tokens max: 162 tokens	min: 40 tokens mean: 258.71 tokens max: 512 tokens

• Samples:

  anchor	positive
  A student developed a database and filed an application for state registration, attaching a description of the database, an abstract, and a statement indicating himself as the right - holder and author. In the application, he indicated two different databases. Does the student's application meet the requirements of the law article?	1. The right - holder can, at his discretion, register such a program or database with the federal executive body for intellectual property during the term of the exclusive right to a computer program or database. Computer programs and databases containing information constituting state secrets are not subject to state registration. The person filing the application for state registration (the applicant) is responsible for disclosing information about computer programs and databases containing information constituting state secrets in accordance with the legislation of the Russian Federation. 2. An application for state registration of a computer program or database (registration application) must relate to one computer program or one database. The registration application must contain: a statement for state registration of a computer program or database indicating...
  Suppose an inheritance agreement provides for the transfer of ownership of a house to A, but A refuses the inheritance. Can B, the other party to the agreement, demand the execution of the agreement regarding the house if the agreement did not contain conditions for transferring the house to B in case of A's refusal?	1. The heir has the right to conclude an agreement with any of the persons who may be called to inherit (Article 1116), the terms of which determine the circle of heirs and the order of transfer of rights to the property of the testator after his death to the surviving parties to the agreement or to surviving third parties who may be called to inherit (inheritance agreement). The inheritance agreement may also contain a condition regarding the soul - executor and impose on the persons participating in the inheritance agreement who may be called to inherit the obligation to perform certain non - contrary - to - law actions of a property or non - property nature, including to execute testamentary refusals or testamentary burdens. The consequences provided for by the inheritance agreement may be made dependent on the circumstances that have occurred by the day of the opening of the inheritance, regarding which it was unknown at the conclusion of the inheritance agreement whether they would occur or not, in particular...
  What is the difference in the procedure for challenging a patent depending on the nature of the violation specified in paragraph 1 of this article?	2. A patent for an invention, utility model, or industrial design can be challenged during its term, established by paragraphs 1 - 3 of Article 1363 of this Code, by filing an objection with the federal executive body for intellectual property by any person who has learned of the violations provided for in sub - paragraphs 1 - 4 of paragraph 1 of this article. A patent for an invention, utility model, or industrial design can be challenged in court by any person who has learned of the violations provided for in sub - paragraph 5 of paragraph 1 of this article. A patent for an invention, utility model, or industrial design can be challenged by an interested person even after the expiration of its term on the grounds and in the manner established by the first and second paragraphs of this paragraph.

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non - Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 2
• per_device_eval_batch_size: 2
• gradient_accumulation_steps: 16
• learning_rate: 2e - 05
• num_train_epochs: 4
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• bf16: True
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

Training Logs

Framework Versions

• Python: 3.11.11
• Sentence Transformers: 3.4.1
• Transformers: 4.43.0
• PyTorch: 2.6.0+cu124
• Accelerate: 1.3.0
• Datasets: 3.3.2
• Tokenizers: 0.19.1

📄 License

This model is licensed under the apache - 2.0 license.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard - Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al - Rfou and Brian Strope and Yun - hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}