Spydaz Web AI Llava

🚀 SpydazWeb AI Model

This project presents a multi - modal and multi - task AI model. It is trained on diverse datasets and is designed to handle a wide range of tasks, from general question - answering to specialized medical and coding tasks.

📚 Documentation

Base Models

The model is based on the following base models:

• LeroyDyer/SpydazWeb_AI_CyberTron_Ultra_7b
• LeroyDyer/LCARS_AI_StarTrek_Computer
• LeroyDyer/_Spydaz_Web_AI_ActionQA_Project
• LeroyDyer/_Spydaz_Web_AI_ChatML_512K_Project
• LeroyDyer/SpyazWeb_AI_DeepMind_Project
• LeroyDyer/SpydazWeb_AI_Swahili_Project
• LeroyDyer/_Spydaz_Web_AI_08
• LeroyDyer/_Spydaz_Web_AI_ChatQA_001
• LeroyDyer/_Spydaz_Web_AI_ChatQA_001_SFT
• LeroyDyer/_Spydaz_Web_AI_Llava

Library Name

The library used for this project is transformers.

Supported Languages

The model supports the following languages:

• en
• sw
• ig
• so
• es
• ca
• xh
• zu
• ha
• tw
• af
• hi
• bm
• su

Datasets

The model has been trained on the following datasets:

• gretelai/synthetic_text_to_sql
• HuggingFaceTB/cosmopedia
• teknium/OpenHermes - 2.5
• Open - Orca/SlimOrca
• Open - Orca/OpenOrca
• cognitivecomputations/dolphin - coder
• databricks/databricks - dolly - 15k
• yahma/alpaca - cleaned
• uonlp/CulturaX
• mwitiderrick/SwahiliPlatypus
• swahili
• Rogendo/English - Swahili - Sentence - Pairs
• ise - uiuc/Magicoder - Evol - Instruct - 110K
• meta - math/MetaMathQA
• abacusai/ARC_DPO_FewShot
• abacusai/MetaMath_DPO_FewShot
• abacusai/HellaSwag_DPO_FewShot
• HaltiaAI/Her - The - Movie - Samantha - and - Theodore - Dataset
• HuggingFaceFW/fineweb
• occiglot/occiglot - fineweb - v0.5
• omi - health/medical - dialogue - to - soap - summary
• keivalya/MedQuad - MedicalQnADataset
• ruslanmv/ai - medical - dataset
• Shekswess/medical_llama3_instruct_dataset_short
• ShenRuililin/MedicalQnA
• virattt/financial - qa - 10K
• PatronusAI/financebench
• takala/financial_phrasebank
• Replete - AI/code_bagel
• athirdpath/DPO_Pairs - Roleplay - Alpaca - NSFW
• IlyaGusev/gpt_roleplay_realm
• rickRossie/bluemoon_roleplay_chat_data_300k_messages
• jtatman/hypnosis_dataset
• Hypersniper/philosophy_dialogue
• Locutusque/function - calling - chatml
• bible - nlp/biblenlp - corpus
• DatadudeDev/Bible
• Helsinki - NLP/bible_para
• HausaNLP/AfriSenti - Twitter
• aixsatoshi/Chat - with - cosmopedia
• xz56/react - llama
• BeIR/hotpotqa
• YBXL/medical_book_train_filtered

Tags

The following tags are associated with the project:

• mergekit
• merge
• Mistral_Star
• Mistral_Quiet
• Mistral
• Mixtral
• Question - Answer
• Token - Classification
• Sequence - Classification
• SpydazWeb - AI
• chemistry
• biology
• legal
• code
• climate
• medical
• LCARS_AI_StarTrek_Computer
• text - generation - inference
• chain - of - thought
• tree - of - knowledge
• forest - of - thoughts
• visual - spacial - sketchpad
• alpha - mind
• knowledge - graph
• entity - detection
• encyclopedia
• wikipedia
• stack - exchange
• Reddit
• Cyber - series
• MegaMind
• Cybertron
• SpydazWeb
• Spydaz
• LCARS
• star - trek
• mega - transformers
• Mulit - Mega - Merge
• Multi - Lingual
• Afro - Centric
• African - Model
• Ancient - One

Pipeline Tag

The pipeline tag for this model is image - to - text.

Quote for Motivation

"Success comes from defining each task in achievable steps. Every completed step is a success that brings you closer to your goal. If your steps are unreachable, failure is inevitable. Winners create more winners, while losers do the opposite. Success is a game of winners!" — # Leroy Dyer (1972 - Present)

LLaVa Model Introduction

LLaVa is an open - source chatbot trained by fine - tuning LlamA/Vicuna on GPT - generated multimodal instruction - following data. It is an auto - regressive language model, based on the transformer architecture. In other words, it is a multi - modal version of LLMs fine - tuned for chat / instructions.

The LLaVa model was proposed in Visual Instruction Tuning and improved in Improved Baselines with Visual Instruction Tuning by Haotian Liu, Chunyuan Li, Yuheng Li and Yong Jae Lee.

The abstract from the paper is the following:

Large multimodal models (LMM) have recently shown encouraging progress with visual instruction tuning. In this note, we show that the fully - connected vision - language cross - modal connector in LLaVA is surprisingly powerful and data - efficient. With simple modifications to LLaVA, namely, using CLIP - ViT - L - 336px with an MLP projection and adding academic - task - oriented VQA data with simple response formatting prompts, we establish stronger baselines that achieve state - of - the - art across 11 benchmarks. Our final 13B checkpoint uses merely 1.2M publicly available data, and finishes full training in ∼1 day on a single 8 - A100 node. We hope this can make state - of - the - art LMM research more accessible. Code and model will be publicly available.

Usage Example

from PIL import Image
import requests
from transformers import AutoProcessor, LlavaForConditionalGeneration

model = LlavaForConditionalGeneration.from_pretrained("llava - hf/llava - 1.5 - 7b - hf")
processor = AutoProcessor.from_pretrained("llava - hf/llava - 1.5 - 7b - hf")

prompt = "USER: <image>\nWhat's the content of the image? ASSISTANT:"
url = "https://www.ilankelman.org/stopsigns/australia.jpg"
image = Image.open(requests.get(url, stream = True).raw)

inputs = processor(text = prompt, images = image, return_tensors = "pt")

# Generate
generate_ids = model.generate(**inputs, max_new_tokens = 15)
processor.batch_decode(generate_ids, skip_special_tokens = True, clean_up_tokenization_spaces = False)[0]

🔧 Technical Details

Training Regimes

The model is trained under the following regimes:

• Alpaca
• ChatML / OpenAI / MistralAI
• Text Generation
• Question/Answer (Chat)
• Planner
• Instruction/Input/Response (instruct)
• Mistral Standard Prompt
• Translation Tasks
• Entitys / Topic detection
• Book recall
• Coding challenges, Code Feedback, Code Sumarization, Commenting Code, code planning and explanation: Software generation tasks
• Agent Ranking and response anyalisis
• Medical tasks
  • PubMed
  • Diagnosis
  • Psychaitry
  • Counselling
  • Life Coaching
  • Note taking
  • Medical smiles
  • Medical Reporting
• Virtual laboritys simulations
• Chain of thoughts methods
• One shot / Multi shot prompting tasks

General Internal Methods

The model is trained for multi - task operations as well as rag and function calling. It is a fully functioning model and is fully uncensored. It has been trained on multiple datasets on the Huggingface hub and kaggle, with the main focus on methodology such as:

• Chain of thoughts
• step by step planning
• tree of thoughts
• forest of thoughts
• graph of thoughts
• agent generation: Voting, ranking, ... dual agent response generation

Training Philosophy

Enhanced Contextual Understanding

Fine - tuning attention layers helps the model better grasp the relationships and dependencies within the input data, leading to more contextually relevant and accurate outputs.

Improved Control over Generation

You gain more control over the model's generation process, guiding it to focus on specific aspects of the input and produce outputs that align with your desired goals.

More Creative and Diverse Outputs

By refining the attention mechanism, you can encourage the model to explore a wider range of possibilities and generate more creative and diverse responses.

Reduced Overfitting

Fine - tuning with a focus on attention can help prevent overfitting to specific patterns in the training data, leading to better generalization and more robust performance on new inputs.

The author's personal training methods are unconventional. They prioritize creating conversations that allow the model to learn new topics from diverse perspectives. Role - playing and conversational training are effective strategies to help the model learn to communicate naturally.

Tokenizer

Definition

A tokenizer is a tool that breaks down text into individual pieces or "tokens" for analysis. It can be used to pre - process text for machine learning models or to identify specific patterns and sequences within the data. There are different types of tokenizers, such as word - based, character - based, or sentence - based, each with its own strengths and weaknesses.

Word - based tokenizers split text into individual words, character - based tokenizers divide text into individual characters, while sentence - based tokenizers break text into sentences. Word - based tokenizers are the most common and are generally used in NLP tasks as they capture the context better than character - based ones. Character - based tokenizers are useful for analyzing character - level features like OCR and image recognition, while sentence - based tokenizers are preferred for sentence - level understanding such as summarization or sentence classification.

Tokenizers can also be customized to suit specific tasks by training them on specific datasets, allowing them to identify specific words or phrases that are relevant to a particular task. This makes them flexible tools for various applications.

In summary, a tokenizer is essential for pre - processing text data for machine learning models and understanding complex language patterns, enabling accurate classification, retrieval, and analysis.

Usage

To use a tokenizer in a machine learning workflow:

1. Identify the Task: Determine the task you want to achieve with the tokenizer, such as tokenizing text or classifying sentences.
2. Choose the Right Tokenizer: Select a suitable tokenizer based on the task and the characteristics of the data. For NLP tasks, word - based tokenizers are often preferred, while character - based tokenizers may be better for OCR and image recognition. Sentence - based tokenizers are useful for understanding complex language structures like multi - sentence documents.
3. Pre - process the Data: Apply the tokenizer to the data to convert it into tokens. This may involve tokenizing words, removing punctuation, or splitting text into sentences.
4. Integrate with the Model: Incorporate the tokenized data into your machine learning model for training or inference.
5. Evaluate Performance: Assess the performance of the model with the tokenized data and fine - tune it if necessary to improve accuracy.
6. Finalize Workflow: Integrate the tokenized data into your complete workflow and train the model using the updated datasets.

History

The concept of tokenization has evolved over time. Early approaches focused on simple character - level segmentation before advancing to word - based approaches in the 1960s. Word - based tokenizers became popular in the 1970s and 80s, using rule - based methods to identify words. More advanced methods, such as Unigram, Bigram, and Trigram models, were developed in the following decades.

In the late 20th century, character - based tokenizers gained attention due to their ability to handle non - word characters like digits and punctuation. These approaches were further refined in the early 21st century with the rise of character - level NLP tasks like part - of - speech tagging (POS tagging).

Modern tokenizers, particularly those used in large language models like GPT - 3, employ subword tokens to capture fine - grained distinctions between words while maintaining efficiency. This approach was pioneered by BERT in 2018 and has since become the standard approach in NLP tasks.

Key Concepts

• Word Tokenization: Splitting text into individual words during pre - processing.
• Character - Based Tokenization: Breaking down text into individual characters for analysis.
• Sentence Tokenization: Dividing text into sentences, ensuring accurate understanding.
• Subword Tokens: Representing words as a combination of subcharacters to capture fine - grained distinctions.
• Rule - Based Tokenization: Identifying words or phrases based on predefined rules and patterns.
• Historical Approaches: Early methods focused on character - level segmentation without considering word boundaries.
• Context Awareness: Recognizing words in context, improving accuracy over historical methods.
• Subword Models: Representing words as a combination of subcharacters to handle out - of - vocabulary (OOO) words during inference.
• Efficiency: Tokenizers optimized for speed and memory usage while maintaining accuracy.

Applications

Tokenization is essential in various NLP tasks.