Wikichat-v2 Open-Source Conversation Model - Free Support for English Text Generation Based on Multiple Datasets

Wikichat V2

Developed by leafspark

WikiChat-v0.2 is a dialogue model currently under training, based on OpenOrca GPT-4 data, cosmopedia, and dolly15k datasets, supporting English text generation tasks.

Large Language Model EnglishOpen Source License:Apache-2.0 #Long-context dialogue #Alpaca instruction fine-tuning #Lightweight LLM

Downloads 86

Release Time : 4/24/2024

Model Overview

WikiChat-v0.2 is a dialogue model based on the Alpaca-style prompt format, primarily used for text generation tasks with support for long-context processing.

Model Features

Long-context support

Supports context lengths of 4096/8192/16384, adjustable via RoPE scaling.

Alpaca-style prompt format

Uses the instruction, input, and response format to construct prompts for optimal performance.

Multi-dataset training

Combines OpenOrca GPT-4 data, cosmopedia, and dolly15k datasets for training to enhance model diversity.

Model Capabilities

Text generation

Dialogue response

Instruction following

Use Cases

Dialogue systems

Q&A systems

Used to answer user questions, such as mathematical calculations, fact queries, etc.

Sample output shows the model can correctly answer simple math problems.

Instruction following

Task execution

Generates corresponding responses or executes tasks based on user instructions.

🚀 WikiChat-v0.2

A model currently in training for having conversations. The uploaded GGUFs are in full FP32 precision, utilizing OpenOrca GPT - 4 data, Cosmopedia for additional data, and Dolly15k for instruction.

🚀 Quick Start

The model is designed for text - generation tasks. To achieve optimal performance, structure your prompts in an instruct format as described below.

✨ Features

Utilizes multiple datasets including HuggingFaceTB/cosmopedia, databricks/databricks - dolly - 15k, and Open - Orca/OpenOrca.
Supports different context lengths: 4096/8192/16384 (with 2/4x RoPE scaling).
Trained with a specific set of hyperparameters to balance training speed and model size.

📦 Installation

No installation steps are provided in the original document.

💻 Usage Examples

Basic Usage

Structure your prompts according to the Alpaca prompt format:

Instruction: {system}
Input: {prompt}
Response: {response}

Example Output

User: What is the square root of 4?

Assistant: The square root of 4 is 2.

📚 Documentation

Model Details

Property	Details
Model Type	Text - generation
Parameters	83.59M (83591800)
Attention Heads	8
Layers	40
Embeddings Size	384
Context	4096/8192/16384 (use 2/4x RoPE scaling, may train a 16k finetuned version later)
Batch Size	16
Training Method	llama.cpp (train - text - from - scratch)

Training Details

Hardware: 1x RTX 3070 8GB (Infrencing speed: 80tok/s, full GPU offload), 1x Ryzen 3 3700x, 96gb RAM
Iterations: 10
Loss Target: 2.5 to 3.0
Samples per 1M Train Tokens: Approx 480 samples/1M train tokens (>0.0001 epoches)
Training Data: Refer to OpenOrca page

🔧 Technical Details

The model is being trained to test the tokenization of OpenOrca and to find a balance between training speed and model size.

📄 License

This project is licensed under the Apache - 2.0 license.

💡 Usage Tip

The model isn't ready yet. Please use it for testing purposes only, especially for evaluating the tokenization of OpenOrca and the balance between training speed and model size.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご