Chexpert Mimic Cxr Impression Baseline

🚀 Transformers图像转文本模型

本项目是一个图像转文本的模型，可用于处理胸部X光图像并生成对应的文本描述，在图像识别和医学影像处理领域具有重要价值。

🚀 快速开始

你可以通过以下步骤快速使用本模型：

1. 确保你已经安装了必要的库，如torch、PIL、transformers和requests。
2. 运行以下代码进行推理：

import torch
from PIL import Image
from transformers import BertTokenizer, ViTImageProcessor, VisionEncoderDecoderModel, GenerationConfig
import requests

mode = "impression"
# Model
model = VisionEncoderDecoderModel.from_pretrained(f"IAMJB/chexpert-mimic-cxr-{mode}-baseline").eval()
tokenizer = BertTokenizer.from_pretrained(f"IAMJB/chexpert-mimic-cxr-{mode}-baseline")
image_processor = ViTImageProcessor.from_pretrained(f"IAMJB/chexpert-mimic-cxr-{mode}-baseline")
#
# Dataset
generation_args = {
   "bos_token_id": model.config.bos_token_id,
   "eos_token_id": model.config.eos_token_id,
   "pad_token_id": model.config.pad_token_id,
   "num_return_sequences": 1,
   "max_length": 128,
   "use_cache": True,
   "beam_width": 2,
}
#
# Inference
refs = []
hyps = []
with torch.no_grad():
   url = "https://huggingface.co/IAMJB/interpret-cxr-impression-baseline/resolve/main/effusions-bibasal.jpg"
   image = Image.open(requests.get(url, stream=True).raw)
   pixel_values = image_processor(image, return_tensors="pt").pixel_values
   # Generate predictions
   generated_ids = model.generate(
       pixel_values,
       generation_config=GenerationConfig(
           **{**generation_args, "decoder_start_token_id": tokenizer.cls_token_id})
   )
   generated_texts = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
   print(generated_texts)

💻 使用示例

基础用法

import torch
from PIL import Image
from transformers import BertTokenizer, ViTImageProcessor, VisionEncoderDecoderModel, GenerationConfig
import requests

mode = "impression"
# Model
model = VisionEncoderDecoderModel.from_pretrained(f"IAMJB/chexpert-mimic-cxr-{mode}-baseline").eval()
tokenizer = BertTokenizer.from_pretrained(f"IAMJB/chexpert-mimic-cxr-{mode}-baseline")
image_processor = ViTImageProcessor.from_pretrained(f"IAMJB/chexpert-mimic-cxr-{mode}-baseline")
#
# Dataset
generation_args = {
   "bos_token_id": model.config.bos_token_id,
   "eos_token_id": model.config.eos_token_id,
   "pad_token_id": model.config.pad_token_id,
   "num_return_sequences": 1,
   "max_length": 128,
   "use_cache": True,
   "beam_width": 2,
}
#
# Inference
refs = []
hyps = []
with torch.no_grad():
   url = "https://huggingface.co/IAMJB/interpret-cxr-impression-baseline/resolve/main/effusions-bibasal.jpg"
   image = Image.open(requests.get(url, stream=True).raw)
   pixel_values = image_processor(image, return_tensors="pt").pixel_values
   # Generate predictions
   generated_ids = model.generate(
       pixel_values,
       generation_config=GenerationConfig(
           **{**generation_args, "decoder_start_token_id": tokenizer.cls_token_id})
   )
   generated_texts = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
   print(generated_texts)

高级用法

# 这里可以根据不同的需求调整模型参数，例如修改生成文本的最大长度、改变束搜索的宽度等。
# 以下是一个修改最大长度和束搜索宽度的示例：
import torch
from PIL import Image
from transformers import BertTokenizer, ViTImageProcessor, VisionEncoderDecoderModel, GenerationConfig
import requests

mode = "impression"
# Model
model = VisionEncoderDecoderModel.from_pretrained(f"IAMJB/chexpert-mimic-cxr-{mode}-baseline").eval()
tokenizer = BertTokenizer.from_pretrained(f"IAMJB/chexpert-mimic-cxr-{mode}-baseline")
image_processor = ViTImageProcessor.from_pretrained(f"IAMJB/chexpert-mimic-cxr-{mode}-baseline")
#
# Dataset
generation_args = {
   "bos_token_id": model.config.bos_token_id,
   "eos_token_id": model.config.eos_token_id,
   "pad_token_id": model.config.pad_token_id,
   "num_return_sequences": 1,
   "max_length": 256,  # 修改最大长度为256
   "use_cache": True,
   "beam_width": 5,  # 修改束搜索宽度为5
}
#
# Inference
refs = []
hyps = []
with torch.no_grad():
   url = "https://huggingface.co/IAMJB/interpret-cxr-impression-baseline/resolve/main/effusions-bibasal.jpg"
   image = Image.open(requests.get(url, stream=True).raw)
   pixel_values = image_processor(image, return_tensors="pt").pixel_values
   # Generate predictions
   generated_ids = model.generate(
       pixel_values,
       generation_config=GenerationConfig(
           **{**generation_args, "decoder_start_token_id": tokenizer.cls_token_id})
   )
   generated_texts = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
   print(generated_texts)

📚 详细文档

你可以参考 在chexpert-plus上的评估 获取更多详细信息。

📄 许可证

本项目采用MIT许可证。

📝 引用说明

如果你使用了本模型，请引用以下文献：

@misc{chambon2024chexpertplusaugmentinglarge,
      title={CheXpert Plus: Augmenting a Large Chest X-ray Dataset with Text Radiology Reports, Patient Demographics and Additional Image Formats}, 
      author={Pierre Chambon and Jean-Benoit Delbrouck and Thomas Sounack and Shih-Cheng Huang and Zhihong Chen and Maya Varma and Steven QH Truong and Chu The Chuong and Curtis P. Langlotz},
      year={2024},
      eprint={2405.19538},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2405.19538}, 
}