Phikon V2

🚀 Phikon-v2 模型卡片

Phikon-v2 是一個基於視覺變換器（Vision Transformer）的大型預訓練模型。它採用 Dinov2 自監督方法，在 PANCAN-XL 數據集上進行預訓練。PANCAN-XL 數據集包含 4.5 億張 20 倍放大的組織學圖像，這些圖像從 6 萬張全切片圖像（WSI）中採樣得到。PANCAN-XL 僅整合了公開可用的數據集，包括用於惡性組織的 CPTAC（6,193 張 WSI）和 TCGA（29,502 張 WSI），以及用於正常組織的 GTEx（13,302 張 WSI）。

與我們之前基於 iBOT 在來自 TCGA（6k WSI）的 4000 萬張組織學圖像上預訓練的基礎模型 Phikon 相比，Phikon-v2 在各種為生物標誌物發現量身定製的弱監督任務上表現更優。為避免與 PANCAN-XL 預訓練數據集發生數據汙染，Phikon-v2 在外部隊列上進行評估，並與一系列表徵學習和基礎模型進行了基準對比。

🚀 快速開始

模型描述

• 開發者：Owkin, Inc
• 模型類型：預訓練視覺骨幹網絡（通過 DINOv2 實現的 ViT-L/16）
• 預訓練數據集：PANCAN-XL，源自公共組織學數據集（TCGA、CPTAC、GTEx、TCIA 等）
• 論文：Arxiv
• 許可證：Owkin 非商業許可證

如何使用（特徵提取）

以下代碼片段展示瞭如何使用 Phikon-v2（CLS 標記）從組織學圖像中提取特徵。這些特徵可用於下游應用，如感興趣區域（ROI）分類（通過線性或 KNN 探測）、切片分類（通過多實例學習）、分割（例如通過 ViT-Adapter）等。

from PIL import Image
import torch
from transformers import AutoImageProcessor, AutoModel


# Load an image
image = Image.open(
    requests.get(
        "https://github.com/owkin/HistoSSLscaling/blob/main/assets/example.tif?raw=true",
        stream=True
    ).raw
)

# Load phikon-v2
processor = AutoImageProcessor.from_pretrained("owkin/phikon-v2")
model = AutoModel.from_pretrained("owkin/phikon-v2")
model.eval()

# Process the image
inputs = processor(image, return_tensors="pt")

# Get the features
with torch.inference_mode():
    outputs = model(**inputs)
    features = outputs.last_hidden_state[:, 0, :]  # (1, 1024) shape

assert features.shape == (1, 1024)

直接使用（使用預提取和凍結的特徵）

Phikon-v2 可以在不同的下游應用中進行微調或不進行微調使用。例如，可以使用多實例學習算法（如 ABMIL）進行切片分類。

下游使用（微調）

你可以在切片級別的下游任務上微調該模型。這個 Colab 筆記本 允許你通過 Hugging Face API 使用 LoRA 微調 Phikon 和 Phikon-v2。

✨ 主要特性

• 強大的特徵提取能力：基於 Vision Transformer 架構和 Dinov2 自監督方法，能夠從組織學圖像中提取高質量特徵。
• 廣泛的數據集支持：在大規模的 PANCAN-XL 數據集上預訓練，該數據集整合了多個公開可用的組織學數據集。
• 優秀的下游任務表現：在各種弱監督任務上優於之前的模型，適用於生物標誌物發現等應用。

📦 安裝指南

軟件依賴

Python 包

• torch>==2.0.0：https://pytorch.org
• torchvision>=0.15.0：https://pytorch.org/vision/stable/index.html
• xformers>=0.0.18：https://github.com/facebookresearch/xformers

代碼倉庫

• DINOv2（自監督學習）：https://github.com/facebookresearch/dinov2

💻 使用示例

基礎用法

from PIL import Image
import torch
from transformers import AutoImageProcessor, AutoModel


# Load an image
image = Image.open(
    requests.get(
        "https://github.com/owkin/HistoSSLscaling/blob/main/assets/example.tif?raw=true",
        stream=True
    ).raw
)

# Load phikon-v2
processor = AutoImageProcessor.from_pretrained("owkin/phikon-v2")
model = AutoModel.from_pretrained("owkin/phikon-v2")
model.eval()

# Process the image
inputs = processor(image, return_tensors="pt")

# Get the features
with torch.inference_mode():
    outputs = model(**inputs)
    features = outputs.last_hidden_state[:, 0, :]  # (1, 1024) shape

assert features.shape == (1, 1024)

高級用法

你可以使用 這個 Colab 筆記本 通過 Hugging Face API 使用 LoRA 微調 Phikon 和 Phikon-v2。

📚 詳細文檔

訓練詳情

• 訓練數據：PANCAN-XL，一個由 4.56 億張 [224×224]、20 倍分辨率的組織學圖像組成的預訓練數據集，這些圖像從 6 萬張 H&E WSI 中採樣得到。
• 訓練機制：使用 PyTorch-FSDP 混合精度的 fp16。
• 訓練目標：採用 DINOv2 自監督學習方法，包含以下損失函數：
  • 具有多裁剪的 DINO 自蒸餾損失
  • iBOT 掩碼圖像建模損失
  • 對 [CLS] 標記的 KoLeo 正則化
• 訓練時長：10 萬次迭代，批次大小為 4,096
• 模型架構：ViT-Large（0.3B 參數）：補丁大小 16，嵌入維度 1024，16 個頭，多層感知機前饋網絡（MLP FFN）
• 使用的硬件：32×4 塊 Nvidia V100 32GB GPU
• 訓練總時長：約 4300 GPU 小時（總計 33 小時）
• 訓練平臺：法國超級集群 Jean-Zay

第三方許可證

視覺變換器架構源自 facebookresearch/dino（Apache 許可證 2.0）和 huggingface/pytorch-image-models（Apache 許可證 2.0）。此代碼基於 DINOv2 代碼倉庫（Apache 許可證 2.0）構建。

預訓練數據集許可證

🔧 技術細節

Phikon-v2 基於 Vision Transformer 架構，利用 Dinov2 自監督學習方法在大規模組織學數據集上進行預訓練。通過結合多種損失函數，如 DINO 自蒸餾損失、iBOT 掩碼圖像建模損失和 KoLeo 正則化，模型能夠學習到魯棒的視覺特徵。在訓練過程中，採用了 PyTorch-FSDP 混合精度訓練，以提高訓練效率。

📄 許可證

本模型使用 Owkin 非商業許可證。

聯繫信息

如有任何額外問題或建議，請聯繫 Alexandre Filiot (alexandre.filiot@owkin.com)。

如何引用

@misc{filiot2024phikonv2largepublicfeature,
      title={Phikon-v2, A large and public feature extractor for biomarker prediction}, 
      author={Alexandre Filiot and Paul Jacob and Alice Mac Kain and Charlie Saillard},
      year={2024},
      eprint={2409.09173},
      archivePrefix={arXiv},
      primaryClass={eess.IV},
      url={https://arxiv.org/abs/2409.09173}, 
}

致謝

我們感謝 DINOv2 的作者們做出的傑出貢獻 [1]。

計算資源

本研究獲得了 IDRIS 高性能計算資源的支持，該支持由 GENCI 分配（2023 - A0141012519）。

數據集

本研究部分結果基於 TCGA 研究網絡生成的數據：https://www.cancer.gov/tcga。基因型 - 組織表達（GTEx）項目由美國國立衛生研究院院長辦公室共同基金以及 NCI、NHGRI、NHLBI、NIDA、NIMH 和 NINDS 支持。本研究中分析使用的數據於 2023 年 7 月 1 日從 GTEx 門戶獲取。

參考文獻

1. Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El - Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P. - Y., Li, S. - W., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jegou, H., Mairal, J., Labatut, P., Joulin, A., & Bojanowski, P. (2024). Dinov2: Learning robust visual features without supervision. arXiv.
2. Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., & Prior, F. (2013). The Cancer Imaging Archive (TCIA): Maintaining and operating a public information repository. Journal of Digital Imaging, 26(6), 1045–1057. Springer Science and Business Media LLC. https://doi.org/10.1007/s10278 - 013 - 9622 - 7
3. National Cancer Institute Clinical Proteomic Tumor Analysis Consortium (CPTAC). (2019). The Clinical Proteomic Tumor Analysis Consortium Acute Myeloid Leukemia Collection (CPTAC - AML) (Version 4) [Data set]. The Cancer Imaging Archive. https://doi.org/10.7937/TCIA.2019.B6FOE619
4. National Cancer Institute Clinical Proteomic Tumor Analysis Consortium (CPTAC). (2018). The Clinical Proteomic Tumor Analysis Consortium Glioblastoma Multiforme Collection (CPTAC - GBM) (Version 15) [Data set]. The Cancer Imaging Archive. https://doi.org/10.7937/K9/TCIA.2018.3RJE41Q1
5. National Cancer Institute Clinical Proteomic Tumor Analysis Consortium (CPTAC). (2020). The Clinical Proteomic Tumor Analysis Consortium Breast Invasive Carcinoma Collection (CPTAC - BRCA) (Version 1) [Data set]. The Cancer Imaging Archive. https://doi.org/10.7937/TCIA.CAEM - YS80
6. National Cancer Institute Clinical Proteomic Tumor Analysis Consortium (CPTAC). (2020). The Clinical Proteomic Tumor Analysis Consortium Colon Adenocarcinoma Collection (CPTAC - COAD) (Version 1) [Data set]. The Cancer Imaging Archive. https://doi.org/10.7937/TCIA.YZWQ - ZZ63
7. National Cancer Institute Clinical Proteomic Tumor Analysis Consortium (CPTAC). (2018). The Clinical Proteomic Tumor Analysis Consortium Head and Neck Squamous Cell Carcinoma Collection (CPTAC - HNSCC) (Version 16) [Data set]. The Cancer Imaging Archive. https://doi.org/10.7937/K9/TCIA.2018.UW45NH81
8. National Cancer Institute Clinical Proteomic Tumor Analysis Consortium (CPTAC). (2018). The Clinical Proteomic Tumor Analysis Consortium Clear Cell Renal Cell Carcinoma Collection (CPTAC - CCRCC) (Version 13) [Data set]. The Cancer Imaging Archive. https://doi.org/10.7937/K9/TCIA.2018.OBLAMN27
9. National Cancer Institute Clinical Proteomic Tumor Analysis Consortium (CPTAC). (2018). The Clinical Proteomic Tumor Analysis Consortium Lung Squamous Cell Carcinoma Collection (CPTAC - LSCC) (Version 15) [Data set]. The Cancer Imaging Archive. https://doi.org/10.7937/K9/TCIA.2018.6EMUB5L2
10. National Cancer Institute Clinical Proteomic Tumor Analysis Consortium (CPTAC). (2019). The Clinical Proteomic Tumor Analysis Consortium Sarcomas Collection (CPTAC - SAR) (Version 10) [Data set]. The Cancer Imaging Archive. https://doi.org/10.7937/TCIA.2019.9BT23R95
11. National Cancer Institute Clinical Proteomic Tumor Analysis Consortium (CPTAC). (2020). The Clinical Proteomic Tumor Analysis Consortium Ovarian Serous Cystadenocarcinoma Collection (CPTAC - OV) (Version 3) [Data set]. The Cancer Imaging Archive. https://doi.org/10.7937/TCIA.ZS4A - JD58
12. National Cancer Institute Clinical Proteomic Tumor Analysis Consortium (CPTAC). (2018). The Clinical Proteomic Tumor Analysis Consortium Pancreatic Ductal Adenocarcinoma Collection (CPTAC - PDA) (Version 14) [Data set]. The Cancer Imaging Archive. https://doi.org/10.7937/K9/TCIA.2018.SC20FO18
13. National Cancer Institute Clinical Proteomic Tumor Analysis Consortium (CPTAC). (2018). The Clinical Proteomic Tumor Analysis Consortium Cutaneous Melanoma Collection (CPTAC - CM) (Version 11) [Data set]. The Cancer Imaging Archive. https://doi.org/10.7937/K9/TCIA.2018.ODU24GZE
14. National Cancer Institute Clinical Proteomic Tumor Analysis Consortium (CPTAC). (2019). The Clinical Proteomic Tumor Analysis Consortium Uterine Corpus Endometrial Carcinoma Collection (CPTAC - UCEC) (Version 12) [Data set]. The Cancer Imaging Archive. https://doi.org/10.7937/K9/TCIA.2018.3R3JUISW
15. Cancer Moonshot Biobank. (2022). Cancer Moonshot Biobank – Colorectal Cancer Collection (CMB - CRC) (Version 5) [Data set]. The Cancer Imaging Archive. https://doi.org/10.7937/DJG7 - GZ87
16. Cancer Moonshot Biobank. (2022). Cancer Moonshot Biobank – Melanoma Collection (CMB - MEL) (Version 5) [Data set]. The Cancer Imaging Archive. https://doi.org/10.7937/GWSP - WH72
17. Cancer Moonshot Biobank. (2022). Cancer Moonshot Biobank – Gastroesophageal Cancer Collection (CMB - GEC) (Version 2) [Data set]. The Cancer Imaging Archive. https://doi.org/10.7937/E7KH - R486
18. Cancer Moonshot Biobank. (2022). Cancer Moonshot Biobank – Lung Cancer Collection (CMB - LCA) (Version 5) [Data set]. The Cancer Imaging Archive. https://doi.org/10.7937/3CX3 - S132
19. Cancer Moonshot Biobank. (2022). Cancer Moonshot Biobank – Multiple Myeloma Collection (CMB - MML) (Version 4) [Data set]. The Cancer Imaging Archive. https://doi.org/10.7937/SZKB - SW39
20. Bakas, S., Sako, C., Akbari, H., Bilello, M., Sotiras, A., Shukla, G., Rudie, J. D., Flores Santamaria, N., Fathi Kazerooni, A., Pati, S., Rathore, S., Mamourian, E., Ha, S. M., Parker, W., Doshi, J., Baid, U., Bergman, M., Binder, Z. A., Verma, R., … Davatzikos, C. (2021). Multi - parametric magnetic resonance imaging (mpMRI) scans for de novo Glioblastoma (GBM) patients from the University of Pennsylvania Health System (UPENN - GBM) (Version 2) [Data set]. The Cancer Imaging Archive. https://doi.org/10.7937/TCIA.709X - DN49
21. Martel, A. L., Nofech - Mozes, S., Salama, S., Akbar, S., & Peikari, M. (2019). Assessment of residual breast cancer cellularity after neoadjuvant chemotherapy using digital pathology [Data set]. The Cancer Imaging Archive. https://doi.org/10.7937/TCIA.2019.4YIBTJNO
22. Campanella, G., Hanna, M. G., Brogi, E., & Fuchs, T. J. (2019). Breast metastases to axillary lymph nodes [Data set]. The Cancer Imaging Archive. https://doi.org/10.7937/TCIA.2019.3XBN2JCC
23. Farahmand, S., Fernandez, A. I., Ahmed, F. S., Rimm, D. L., Chuang, J. H., Reisenbichler, E., & Zarringhalam, K. (2022). HER2 and trastuzumab treatment response H&E slides with tumor ROI annotations (Version 3) [Data set]. The Cancer Imaging Archive. https://doi.org/10.7937/E65C - AM96
24. Pataki, B. A., Olar, A., Ribli, D., Pesti, A., Kontsek, E., Gyongyosi, B., Bilecz, A., Kovács, T., Kovács, K. A., Kiss, Z., Szócska, M., Pollner, P., & Csabai, I. (2021). Digital pathological slides from Hungarian (Europe) colorectal cancer screening (Version 2) [Data set]. The Cancer Imaging Archive. https://doi.org/10.7937/TCIA.9CJF - 0127
25. Pennycuick, A., Teixeira, V. H., AbdulJabbar, K., Raza, S. E. A., Lund, T., Akarca, A. U., Rosenthal, R., Kalinke, L., Chandrasekharan, D. P., Pipinikas, C. P., Lee - Six, H., Hynds, R. E., Gowers, K. H. C., Henry, J. Y., Millar, F. R., Hagos, Y. B., Denais, C., Falzon, M., Moore, D. A., Antoniou, S., Durrenberger, P. F., Furness, A. J., Carroll, B., Marceaux, C., Asselin - Labat, M. L., Larson, W., Betts, C., Coussens, L. M., Thakrar, R. M., George, J., Swanton, C., Thirlwell, C., Campbell, P. J., Marafioti, T., Yuan, Y., Quezada, S. A., McGranahan, N., & Janes, S. M. (2020). Immune surveillance in clinical regression of preinvasive squamous cell lung cancer. Cancer Discovery, 10(10), 1489 - 1499. https://doi.org/10.1158/2159 - 8290.CD - 19 - 1366
26. National Lung Screening Trial Research Team. (2013). Data from the National Lung Screening Trial (NLST) [Data set]. The Cancer Imaging Archive. https://doi.org/10.7937/TCIA.HMQ8 - J677
27. Wang, C. - W., Chang, C. - C., Lo, S. - C., Lin, Y. - J., Liou, Y. - A., Hsu, P. - C., Lee, Y. - C., & Chao, T. - K. (2021). A dataset of histopathological whole slide images for classification of treatment effectiveness to ovarian cancer (Ovarian Bevacizumab Response) (Version 2) [Data set]. The Cancer Imaging Archive. https://doi.org/10.7937/TCIA.985G - EY35
28. Chowdhury, S., Kennedy, J. J., Ivey, R. G., Murillo, O., Hosseini, N., Song, X., Petralia, F., Calinawan, A., Voytovich, U. J., Savage, S. R., Berry, A., Reva, B., Ozbek, U., Krek, A., Ma, W., da Veiga Leprevost, F., Ji, J., Yoo, S., Lin, C., … Paulovich, A. G. (2023). Proteogenomic analysis of chemo - refractory high grade serous ovarian cancer (PTRC - HGSOC) (Version 1) [Data set]. The Cancer Imaging Archive. https://doi.org/10.7937/6RDA - P940
29. Hodis, E., Torlai Triglia, E., Kwon, J. Y. H., Biancalani, T., Zakka, L. R., Parkar, S., Hütter, J. C., Buffoni, L., Delorey, T. M., Phillips, D., Dionne, D., Nguyen, L. T., Schapiro, D., Maliga, Z., Jacobson, C. A., Hendel, A., Rozenblatt - Rosen, O., Mihm, M. C. Jr., Garraway, L. A., & Regev, A. (2022). Stepwise - edited, human melanoma models reveal mutations' effect on tumor and microenvironment. Science, 376(6592), eabi8175. https://doi.org/10.1126/science.abi8175