Ppace V1.0

🚀 Pandemic PACT Advanced Categorisation Engine (PPACE)

The Pandemic PACT Advanced Categorisation Engine (PPACE) is an 8B fine - tuned large language model (LLM). It's designed to automatically classify research abstracts from funded biomedical projects according to WHO - aligned research priorities. As part of the GLOPID - R Pandemic PACT project, PPACE helps track and analyze research funding and clinical evidence for various diseases with outbreak potential. The model uses a human - annotated dataset augmented with rationales from a larger LLM, which explain the chosen labels, improving interpretability and accuracy.

🚀 Quick Start

This section provides a step - by - step guide on how to use the PPACE model for classifying biomedical research projects.

✨ Features

• Automated Categorization: Automatically classifies biomedical research abstracts according to WHO - aligned research priorities.
• Interpretability: Leverages rationales generated by a larger LLM to explain the chosen labels.
• Efficient Training: Fine - tuned using Low - Rank Adaptation (LoRA) for efficient training while maintaining high performance.

📦 Installation

The installation process mainly involves setting up the necessary Python libraries and loading the model. You need to have transformers (ts in the code), torch installed in your Python environment.

💻 Usage Examples

Basic Usage

Preparing the input Prompt

def construct_input_prompt(title, abstract):
    categories = """We have a project in the area of biomedical research which we want to classify in terms of the research priorities it related to. We have 12 possible research priorities and a project can be mapped to 1 or more of these priorities. The following is a guide on what each of these 12 categories are alongside the specific areas that they cover. 

1. Pathogen: Natural History, Transmission, and Diagnostics:
    Development of diagnostic tools, understanding pathogen morphology, genomics, and genotyping, studying immunity, using disease models, and assessing the environmental stability of pathogens.

2. Animal and Environmental Research & Research on Diseases Vectors:
    Animal sources, transmission routes, vector biology, and control strategies for vectors.

3. Epidemiological Studies:
    Research on disease transmission dynamics, susceptibility, control measure effectiveness, and disease mapping through surveillance and reporting.

4. Clinical Characterisation and Management in Humans:
    Prognostic factors for disease severity, disease pathogenesis, supportive care and management, long - term health consequences, and clinical trials for disease management.

5. Infection Prevention and Control:
    Research on community restriction measures, barriers and PPE, infection control in healthcare settings, and measures at the human - animal interface.

6. Therapeutics Research, Development, and Implementation:
    Pre - clinical studies for therapeutic development, clinical trials for therapeutic safety and efficacy, development of prophylactic treatments, logistics and supply chain management for therapeutics, clinical trial design for therapeutics, and research on adverse events related to therapeutic administration.

7. Vaccines Research, Development, and Implementation:
    Pre - clinical studies for vaccine development, clinical trials for vaccine safety and efficacy, logistics and distribution strategies for vaccines, vaccine design and administration, clinical trial design for vaccines, research on adverse events related to immunisation, and characterisation of vaccine - induced immunity.

8. Research to Inform Ethical Issues:
    Ethical considerations in research design, ethical issues in public health measures, ethical clinical decision - making, ethical resource allocation, ethical governance, and ethical considerations in social determinants of health.

9. Policies for public health, disease control and  community resilience:
    Approaches to public health interventions, community engagement, communication and infodemic management, vaccine/therapeutic hesitancy, and policy research and interventions.

10. Secondary Impacts of Disease, Response, and Control Measures:
    Indirect health impacts, social impacts, economic impacts, and other secondary impacts such as environmental effects, food security, and infrastructure.

11. Health Systems Research:
    Health service delivery, health financing, access to medicines and technologies, health information systems, health leadership and governance, and health workforce management.

12. Capacity Strengthening:
    Individual capacity building, institutional capacity strengthening, systemic/environmental components, and cross - cutting activities across all levels of capacity building."""

    prompt = "Based on the research categorization guidelines, classify the following project into the appropriate primary research priorities using the categories 1 to 12."
    prompt += f"\n\n{categories.strip()}\n\nProject Information:\n\n"
    prompt += f"### Title:\n'''\n{title.strip()}\n'''\n\n### Abstract:\n'''\n{abstract.strip()}\n'''\n\n"
    prompt += "Based on this information, identify the relevant research categories for this project. Provide clear explanation for your choices. Section your response in the following format:"
    prompt += "\n\n### Explanation: ...\n\n### Categories: ..."
    
    return prompt

title = "Neutralization of Primate Immunodeficiency Viruses"
abstract = "We will repurpose existing assays, techniques and expertise that are central to our project team's virology, structural biology, vaccine development and protein production skill - sets for HIV research, to now also work on SARS - CoV - 2 during the COVID - 19 pandemic emergency. These interactive research efforts will draw on our established methodologies and should represent a productive use of our existing NIH grant resources. We note that there continue to be institutional restrictions at all three performance sites on the effort that can be applied to our original goals relating to HIV - 1 vaccine research and development. Those goals will be unchanged, but will be pursued at a reduced effort during the period when we also work on the new SARS - CoV - 2 projects for which we have fewer institutional restrictions due to the COVID - 19 pandemic."

input_prompt = construct_input_prompt(title, abstract)

Loading the Model

import torch as torch
import transformers as ts

model_id = "nlpie/ppace - v1.0"

tokenizer = ts.AutoTokenizer.from_pretrained(model_id)

tokenizer.padding_side = 'right'
tokenizer.pad_token = tokenizer.eos_token
tokenizer.add_eos_token = True
tokenizer.bos_token, tokenizer.eos_token

model = ts.AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    attn_implementation="flash_attention_2",
    torch_dtype=torch.bfloat16)

Inference

def generateOutput(input):
    inputs = tokenizer(
        [
            tokenizer.decode(tokenizer.apply_chat_template([{"content": input, "role": "user"}]))
        ], return_tensors="pt").to("cuda")

    output = model.generate(**inputs, max_new_tokens=512, num_beams = 4, eos_token_id=tokenizer("<|eot_id|>", add_special_tokens=False)["input_ids"][0])

    return tokenizer.decode(output[0])

output = generateOutput(input_prompt)

Complete Code

import torch as torch
import transformers as ts


def construct_input_prompt(title, abstract):
    categories = """We have a project in the area of biomedical research which we want to classify in terms of the research priorities it related to. We have 12 possible research priorities and a project can be mapped to 1 or more of these priorities. The following is a guide on what each of these 12 categories are alongside the specific areas that they cover. 

1. Pathogen: Natural History, Transmission, and Diagnostics:
    Development of diagnostic tools, understanding pathogen morphology, genomics, and genotyping, studying immunity, using disease models, and assessing the environmental stability of pathogens.

2. Animal and Environmental Research & Research on Diseases Vectors:
    Animal sources, transmission routes, vector biology, and control strategies for vectors.

3. Epidemiological Studies:
    Research on disease transmission dynamics, susceptibility, control measure effectiveness, and disease mapping through surveillance and reporting.

4. Clinical Characterisation and Management in Humans:
    Prognostic factors for disease severity, disease pathogenesis, supportive care and management, long - term health consequences, and clinical trials for disease management.

5. Infection Prevention and Control:
    Research on community restriction measures, barriers and PPE, infection control in healthcare settings, and measures at the human - animal interface.

6. Therapeutics Research, Development, and Implementation:
    Pre - clinical studies for therapeutic development, clinical trials for therapeutic safety and efficacy, development of prophylactic treatments, logistics and supply chain management for therapeutics, clinical trial design for therapeutics, and research on adverse events related to therapeutic administration.

7. Vaccines Research, Development, and Implementation:
    Pre - clinical studies for vaccine development, clinical trials for vaccine safety and efficacy, logistics and distribution strategies for vaccines, vaccine design and administration, clinical trial design for vaccines, research on adverse events related to immunisation, and characterisation of vaccine - induced immunity.

8. Research to Inform Ethical Issues:
    Ethical considerations in research design, ethical issues in public health measures, ethical clinical decision - making, ethical resource allocation, ethical governance, and ethical considerations in social determinants of health.

9. Policies for public health, disease control and  community resilience:
    Approaches to public health interventions, community engagement, communication and infodemic management, vaccine/therapeutic hesitancy, and policy research and interventions.

10. Secondary Impacts of Disease, Response, and Control Measures:
    Indirect health impacts, social impacts, economic impacts, and other secondary impacts such as environmental effects, food security, and infrastructure.

11. Health Systems Research:
    Health service delivery, health financing, access to medicines and technologies, health information systems, health leadership and governance, and health workforce management.

12. Capacity Strengthening:
    Individual capacity building, institutional capacity strengthening, systemic/environmental components, and cross - cutting activities across all levels of capacity building."""

    prompt = "Based on the research categorization guidelines, classify the following project into the appropriate primary research priorities using the categories 1 to 12."
    prompt += f"\n\n{categories.strip()}\n\nProject Information:\n\n"
    prompt += f"### Title:\n'''\n{title.strip()}\n'''\n\n### Abstract:\n'''\n{abstract.strip()}\n'''\n\n"
    prompt += "Based on this information, identify the relevant research categories for this project. Provide clear explanation for your choices. Section your response in the following format:"
    prompt += "\n\n### Explanation: ...\n\n### Categories: ..."

    return prompt


title = "Neutralization of Primate Immunodeficiency Viruses"
abstract = "We will repurpose existing assays, techniques and expertise that are central to our project team's virology, structural biology, vaccine development and protein production skill - sets for HIV research, to now also work on SARS - CoV - 2 during the COVID - 19 pandemic emergency. These interactive research efforts will draw on our established methodologies and should represent a productive use of our existing NIH grant resources. We note that there continue to be institutional restrictions at all three performance sites on the effort that can be applied to our original goals relating to HIV - 1 vaccine research and development. Those goals will be unchanged, but will be pursued at a reduced effort during the period when we also work on the new SARS - CoV - 2 projects for which we have fewer institutional restrictions due to the COVID - 19 pandemic."

input_prompt = construct_input_prompt(title, abstract)

model_id = "nlpie/ppace - v1.0"

tokenizer = ts.AutoTokenizer.from_pretrained(model_id)

tokenizer.padding_side = 'right'
tokenizer.pad_token = tokenizer.eos_token
tokenizer.add_eos_token = True
tokenizer.bos_token, tokenizer.eos_token

model = ts.AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    attn_implementation="flash_attention_2",
    torch_dtype=torch.bfloat16)


def generateOutput(input):
    inputs = tokenizer(
        [
            tokenizer.decode(tokenizer.apply_chat_template([{"content": input, "role": "user"}]))
        ], return_tensors="pt").to("cuda")

    output = model.generate(**inputs, max_new_tokens=512, num_beams = 4, eos_token_id=tokenizer("<|eot_id|>", add_special_tokens=False)["input_ids"][0])

    return tokenizer.decode(output[0])


output = generateOutput(input_prompt)

print(output)

📚 Documentation

Model Details

PPACE is fine - tuned using Low - Rank Adaptation (LoRA) to ensure efficient training while maintaining high performance. The fine - tuning process involves training the model for 4 epochs on a dataset of 5142 projects, using 8 A100 GPUs with a batch size of 1 per GPU and 4 gradient accumulation steps.

Hyperparameters

📄 License

This project is licensed under the MIT License.