Biotrove CLIP

🚀 BioTrove-CLIP

BioTrove-CLIP is a suite of vision-language foundation models for biodiversity, trained on a large - scale dataset and evaluated on zero - shot image classification tasks.

🚀 Quick Start

BioTrove-CLIP is a new suite of vision - language foundation models for biodiversity. These CLIP - style foundation models were trained on [BioTrove - Train](https://huggingface.co/BGLab/BioTrove - Train), a large - scale dataset with 40 million images of 33K species of plants and animals. The models are used for zero - shot image classification tasks.

✨ Features

• Model type: Vision Transformer (ViT - B/16, ViT - L/14)
• License: MIT
• Fine - tuned from model: OpenAI CLIP, MetaCLIP, BioCLIP

📚 Documentation

Model Description

BioTrove - CLIP is based on OpenAI's CLIP model. The models were trained on [BioTrove - Train](https://huggingface.co/BGLab/BioTrove - Train) with the following configurations:

• BioTrove - CLIP - O: A ViT - B/16 backbone initialized from the OpenCLIP's checkpoint was trained for 40 epochs.
• BioTrove - CLIP - B: A ViT - B/16 backbone initialized from the BioCLIP's checkpoint was trained for 8 epochs.
• BioTrove - CLIP - M: A ViT - L/14 backbone initialized from the MetaCLIP's checkpoint was trained for 12 epochs.

To access the checkpoints of the above models, go to the Files and versions tab and download the weights. These weights can be directly used for zero - shot classification and finetuning. The filenames correspond to the specific model weights:

• BioTrove - CLIP - O: biotroveclip - vit - b - 16 - from - openai - epoch - 40.pt
• BioTrove - CLIP - B: biotroveclip - vit - b - 16 - from - bioclip - epoch - 8.pt
• BioTrove - CLIP - M: biotroveclip - vit - l - 14 - from - metaclip - epoch - 12.pt

Model Training

See the Model Training section on the Github for examples of how to use BioTrove - CLIP models in zero - shot image classification tasks.

We train three models using a modified version of the BioCLIP / OpenCLIP codebase. Each model is trained on Arboretum - 40M, on 2 nodes, 8xH100 GPUs, on NYU's [Greene](https://sites.google.com/nyu.edu/nyu - hpc/hpc - systems/greene) high - performance compute cluster. We publicly release all code needed to reproduce our results on the Github page.

We optimize our hyperparameters prior to training with Ray. Our standard training parameters are as follows:

--dataset - type webdataset 
--pretrained openai 
--text_type random 
--dataset - resampled 
--warmup 5000 
--batch - size 4096 
--accum - freq 1 
--epochs 40
--workers 8 
--model ViT - B - 16 
--lr 0.0005 
--wd 0.0004 
--precision bf16 
--beta1 0.98 
--beta2 0.99 
--eps 1.0e - 6 
--local - loss 
--gather - with - grad 
--ddp - static - graph 
--grad - checkpointing

For more extensive documentation of the training process and the significance of each hyperparameter, we recommend referencing the OpenCLIP and BioCLIP documentation, respectively.

Model Validation

For validating the zero - shot accuracy of our trained models and comparing to other benchmarks, we use the VLHub repository with some slight modifications.

Pre - Run

After cloning the Github repository and navigating to the BioTrove/model_validation directory, we recommend installing all the project requirements into a conda container; pip install -r requirements.txt. Also, before executing a command in VLHub, please add BioTrove/model_validation/src to your PYTHONPATH.

export PYTHONPATH="$PYTHONPATH:$PWD/src";

Base Command

A basic BioTrove - CLIP model evaluation command can be launched as follows. This example would evaluate a CLIP - ResNet50 checkpoint whose weights resided at the path designated via the --resume flag on the ImageNet validation set, and would report the results to Weights and Biases.

python src/training/main.py --batch - size = 32 --workers = 8 --imagenet - val "/imagenet/val/" --model = "resnet50" --zeroshot - frequency = 1 --image - size = 224 --resume "/PATH/TO/WEIGHTS.pth" --report - to wandb

Training Links

• Main Dataset Repository: BioTrove
• Dataset Paper: BioTrove: A Large Curated Image Dataset Enabling AI for Biodiversity (arXiv)
• HF Dataset card: [BioTrove - Train (40M)](https://huggingface.co/datasets/BGLab/BioTrove - Train)

Model's Limitation

All the BioTrove - CLIP models were evaluated on the challenging CONFOUNDING - SPECIES benchmark. However, all the models performed at or below random chance. This could be an interesting avenue for follow - up work and further expand the models' capabilities.

In general, we found that models trained on web - scraped data performed better with common names, whereas models trained on specialist datasets performed better when using scientific names. Additionally, models trained on web - scraped data excel at classifying at the highest taxonomic level (kingdom), while models begin to benefit from specialist datasets like [BioTrove - Train (40M)](https://huggingface.co/datasets/BGLab/BioTrove - Train) and [Tree - of - Life - 10M](https://huggingface.co/datasets/imageomics/TreeOfLife - 10M) at the lower taxonomic levels (order and species). From a practical standpoint, BioTrove - CLIP is highly accurate at the species level, and higher - level taxa can be deterministically derived from lower ones.

Addressing these limitations will further enhance the applicability of models like BioTrove - CLIP in real - world biodiversity monitoring tasks.

Acknowledgements

This work was supported by the AI Research Institutes program supported by the NSF and USDA - NIFA under AI Institute: for Resilient Agriculture, Award No. 2021 - 67021 - 35329. This was also partly supported by the NSF under CPS Frontier grant CNS - 1954556. Also, we gratefully acknowledge the support of NYU IT [High Performance Computing](https://www.nyu.edu/life/information - technology/research - computing - services/high - performance - computing.html) resources, services, and staff expertise.

@misc{yang2024arboretumlargemultimodaldataset,
        title={Arboretum: A Large Multimodal Dataset Enabling AI for Biodiversity}, 
        author={Chih - Hsuan Yang, Benjamin Feuer, Zaki Jubery, Zi K. Deng, Andre Nakkab,
           Md Zahid Hasan, Shivani Chiranjeevi, Kelly Marshall, Nirmal Baishnab, Asheesh K Singh,
            Arti Singh, Soumik Sarkar, Nirav Merchant, Chinmay Hegde, Baskar Ganapathysubramanian},
        year={2024},
        eprint={2406.17720},
        archivePrefix={arXiv},
        primaryClass={cs.CV},
        url={https://arxiv.org/abs/2406.17720}, 
  }

For more details and access to the Arboretum dataset, please visit the Project Page.

📄 License

This project is under the MIT license.