Clinical Trials All MiniLM L6 V2

🚀 基于sentence-transformers/all-MiniLM-L6-v2的句子转换器

本项目是一个基于sentence-transformers框架，从sentence-transformers/all-MiniLM-L6-v2微调而来的模型。它能够将句子和段落映射到一个384维的密集向量空间，可用于语义文本相似度计算、语义搜索、释义挖掘、文本分类、聚类等任务。

🚀 快速开始

安装依赖

首先，你需要安装sentence-transformers库：

pip install -U sentence-transformers

运行推理

安装完成后，你可以加载该模型并进行推理：

from sentence_transformers import SentenceTransformer

# 从🤗 Hub下载模型
model = SentenceTransformer("sentence_transformers_model_id")
# 定义待编码的句子
sentences = [
    'open label trial safety efficacy sym001 treatment immune thrombocytopenic purpura itp. If your serious vaccine-induced adverse event has been entered in the CDC Vaccine Adverse Event Reporting System (VAERS) we are interested in enrolling you for this study in order to log your symptoms.\n\nThe primary goal of this study is to create a national database and gather vaccine-associated serious adverse events/injury data from newly vaccinated individuals in the US in order to identify the possible underlying causal relationships and plausible underlying biological mechanisms. The project aims to identify the genetic determinants of vaccine-induced adverse response by studying host genetics. We plan to use whole genome sequencing to identify single nucleotide polymorphisms associated with cardiovascular, neurological, gastrointestinal, musculoskeletal and immunological symptoms induced by vaccine administration.\n\nThe secondary goal is to establish criteria that enable classification of vaccine-induced adverse events/injuries compare data from our database with the official Vaccine Injury Table National Vaccine Injury Compensation Program on or after March 21, 2017.\n\nThe tertiary goal is to establish a database to gather detailed long-term adverse reaction data from subjects enrolled in FDA Emergency Use Authorized vaccine clinical trials.',
    'Immune Thrombocytopenic Purpura inclusion criterion confirm presence thrombocytopenia platelet count   30000mm3 predose visit history isolated itp rhdpositive serology previous treatment response line therapy itpexclusion criterion know clinical picture suggestive cause thrombocytopenia especially systematic lupus erythematosusantiphospholipid syndrome evans syndrome immunodeficiency state lymphoproliferative disorder liver diseaseingestion drug quinidinequinine heparin sulfonamide hereditary thrombocytopenia confirm relevant laboratory finding suspect infection hiv hepatitis c h pylori clinical splenomegaly history abnormal bone marrow examination ongoing haemorrhage correspond grade 3 4 bleeding scale underlie haemolytic condition history splenectomy subject pregnant breast feeding intend pregnant incidence severity adverse event aes include adverse event saes 6 week post dose measurement platelet count day 1 week 6',
    'Multiple System Atrophy inclusion criteriadiagnostic1 participant diagnosis possible probable msa modify gilman et al 2008 diagnostic criteria2 participant onset msa symptom occur 4 year screen assess investigator3 evidence msa specific symptom deficit measure umsars scaleexclusion criteriamedical history1 participant contraindication study proceduresdiagnostic assessments1 presence confound diagnosis andor condition affect participant safety study investigator judgement2 participant participation previous study diseasemodifye therapy prove receipt active treatment compromise interpretability datum present study consultation medical monitor designeeother1 participant participate study investigate active passive immunization αsynuclein αsyn progressive disease pd msa immunoglobulin g therapy 6 month screen change baseline modify unified multiple system atrophy rating scale umsar week 52 umsar historical review 11item scale adapt unify parkinson disease rating scale updrs assess activity relate motor disability relate autonomic dysfunction item score 0 normal 3 severe total score sum score domain range 0 33 high score mean poor health 52 week change baseline 11item umsar week 52 11 item umsar include 11 item ii assess motor autonomic disability umsar historical review assess activity relate motor disability autonomic dysfunction umsar ii motor examination measure functional impairment specific parkinsonian cerebellar feature item score 0 normal 4 severe total score sum score domain range 0 44 high score mean poor health 52 weekschange baseline umsar total score umsar   ii week 52 umsar total scale consist item umsars part ii umsar historical review 12item scale assess activity relate motor disability autonomic dysfunction item score 0 normal 4 severe umsar ii motor examination 14item scale measure functional impairment eg speech rapid alternate movement hand finger tap leg agility select complex movement specific parkinsonian tremor rest cerebellar ocular motor dysfunction heelshin test feature item score 0 normal 4 severe 52 weekschange baseline umsars week 52 umsar historical review modified 11item scale adapt updrs assess activity relate motor disability 8 item 4 novel item relate autonomic dysfunction item score 0 normal 4 severe total score sum score item range 0 44 high score mean poor health 52 weekschange baseline umsars ii week 52 umsar ii motor examination 14item scale item eg speech rapid alternate movement hand finger tap leg agility measure functional impairment select complex movement item directly refer specific parkinsonian tremor rest cerebellar ocular motor dysfunction heelshin test feature motor examination section umsar base modify updrsiii item addition novel item heelkneeshin ataxia item score 0 normal 4 severe total score sum score item range 0 56 high score mean poor health 52 weeksclinical global impressionseverity cgis score cgis assess clinicians impression participant clinical condition clinician use total clinical experience participant population rate current severity participant illness 7point scale range 1 normal ill 7 extremely ill participant high score mean well health 52 weekschange baseline scale outcome parkinson disease   autonomic dysfunction scopaaut total score scopaaut patientreported outcome assess autonomic function autonomic function critical symptom domain msa scale selfcomplete participant consist 25 item assess follow domain gastrointestinal 7 item urinary 6 item cardiovascular 3 item thermoregulatory 4 item pupillomotor 1 item sexual 2 item man 2 item woman score item range 0 experience symptom 3 experience symptom total composite score include domain report score range 0 symptom 69 high burden symptom 52 weeksoverall survival os os define time day study drug administration death cause 52 weekschange baseline level cerebrospinal fluid csf free alphasynuclein αsyn 52 weekscmax maximum observe serum concentration tak341 predose day 1 29 57 85 169 253 337 multiple timepoint 24 hour postdose day 1 57 85 169 337 anytime day 365 427 early termination day 57 applicable early pk cohortstmax time occurrence cmax serum tak341 predose day 1 29 57 85 169 253 337 multiple timepoint 24 hour postdose day 1 57 85 169 337 anytime day 365 427 early termination day 57 applicable early pk cohortsaucτ area concentrationtime curve dose interval serum tak341 predose day 1 29 57 85 169 253 337 multiple timepoint 24 hour postdose day 1 57 85 169 337 anytime day 365 427 early termination day 57 applicable early pk cohortscsf concentration tak341 lumbar puncture csf sampling perform predose day 1 85 applicable early pk cohort 365number participant adverse event ae adverse event ae define untoward medical occurrence participant administer pharmaceutical product untoward medical occurrence necessarily causal relationship treatment datum report number participant analyze safety parameter include clinically significant abnormal value clinical laboratory evaluation vital sign ecg parameters physical examination neurological examination columbiasuicide severity rating scale cssrs 52 weeksnumber participant antidrug antibody 52 week',
]
# 对句子进行编码
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# 计算嵌入向量之间的相似度分数
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

✨ 主要特性

• 语义表示能力：能够将句子和段落映射到一个384维的密集向量空间，从而捕捉文本的语义信息。
• 多任务适用性：可用于语义文本相似度计算、语义搜索、释义挖掘、文本分类、聚类等多种自然语言处理任务。
• 微调灵活性：基于sentence-transformers框架，可在自定义数据集上进行微调，以适应特定的任务需求。

📦 安装指南

安装本模型所需的依赖库，可使用以下命令：

pip install -U sentence-transformers

📚 详细文档

模型详情

模型描述

模型来源

• 文档：Sentence Transformers Documentation
• 仓库：Sentence Transformers on GitHub
• Hugging Face：Sentence Transformers on Hugging Face

完整模型架构

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

训练详情

训练数据集

未命名数据集

• 规模：92,934个训练样本
• 列信息：包含<code>Text1</code>和<code>Text2</code>两列
• 基于前1000个样本的近似统计信息：

  	Text1	Text2
  类型	字符串	字符串
  详情	最小值：29个词元 平均值：104.36个词元 最大值：256个词元	最小值：8个词元 平均值：227.98个词元 最大值：256个词元

• 样本示例：

  Text1	Text2
  study people normal kidney function people reduce kidney function test bi 1467335 process body. The primary objective of the current study is to investigate the influence of moderate renal impairment on the pharmacokinetics of multiple doses in comparison to a matched control group with normal renal function.	Renal Insufficiency
  16w interventional study titration doseefficacy assessment exelon chinese alzheimers disease patient. To investigate the efficacy of Exelon capsule at maximal tolerated dose in mild to moderate Chinese AD patients via dosage titration from 3mg/d to 12mg/d in a 16 weeks duration	Alzheimer's Disease key inclusion criterion diagnosis dementia alzheimers type accord dsmiv criterion clinical diagnosis probable ad accord nincdsadrda criteria mmse score 10 26 treatment naïve patient stop donepezil galantamine huperzine memantine 2 week stable medical condition sign inform consent form patient hisher legal guardiankey exclusion criterion severe ad patient history cerebrovascular disease active uncontrolled epilepsy active hypothyroidism asthma cns infection neurodegenerative disorder advanced severe progressive unstable medical condition attend clinical trial take clinical trial drug score 4 modify hachinski ischemic scale mhis patient achei memantine mean change baseline alzheimer disease assessment scale cognitive subscale adascog alzheimer disease assessment scale cognitive subscale adascog measure change cognitive function alzheimer disease assessment scale adas scale measure specific cognitive behavior disorder alzheimer disease ad patient alzheimer di...
  case series saneso 360 gastroscope. To confirm the procedural performance of the Saneso 360° gastroscope in Esophago-gastro-duodenoscopy (EGD) procedures.	EGD Procedure inclusion criterion 18 74 year age willing able comply study procedure provide write inform consent participate study schedule clinically indicate routine egd procedure asa class 13exclusion criterion alter esophageal gastric duodenal anatomy pregnant woman child 18 year age adult 75 year age subject routine endoscopic procedure contraindicate comorbid medical condition patient currently enrol investigational study directly interfere current study prior write approval sponsor asa class 45 successful egd procedure success assess end procedure 1 procedure success define successful intubation portion duodenum photograph portion duodenum take 24 hour study day endoscopist qualitative rating saneso 360 gastroscope endoscopist rate experience saneso 360 gastroscope immediately follow completion study procedure 1 fivepoint likert scale 5 excellent 4 good 3 acceptable 2 difficult 1unacceptable 24 hour study dayendoscopist qualitative rating saneso 360 gastroscope compare past...

• 损失函数：使用MatryoshkaLoss，参数如下：

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          384,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

训练超参数

非默认超参数

• per_device_train_batch_size：16
• learning_rate：2e-05
• num_train_epochs：1
• warmup_ratio：0.1
• fp16：True
• batch_sampler：no_duplicates

所有超参数

训练日志

框架版本

• Python：3.10.12
• Sentence Transformers：3.3.1
• Transformers：4.48.1
• PyTorch：2.1.0a0+32f93b1
• Accelerate：1.3.0
• Datasets：3.2.0
• Tokenizers：0.21.0

📄 许可证

文档中未提及许可证相关信息。

🔧 技术细节

本模型基于sentence-transformers框架，从sentence-transformers/all-MiniLM-L6-v2微调而来。模型结构包含一个Transformer层、一个池化层和一个归一化层。在训练过程中，使用了MatryoshkaLoss损失函数，并设置了一系列超参数进行优化。

📖 引用

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}