AIbase
EN
Home
Xtremedistil L6 H384 Uncased Finetuned Wikitext103
X

Xtremedistil L6 H384 Uncased Finetuned Wikitext103

Developed by saghar
This model is a lightweight distilled version based on Microsoft's Xtremedistil, fine-tuned on the wikitext dataset, suitable for text generation tasks.
Large Language Model
Transformers
Open Source License:MIT#Text Generation Fine-tuning#Lightweight Distillation#WikiText Adaptation
Downloads 18
Release Time : 3/19/2022

Model Overview

This is a fine-tuned lightweight language model based on Microsoft's Xtremedistil architecture, specifically optimized for wikitext data, and can be used for text generation and related natural language processing tasks.

Model Features

Lightweight Architecture
Adopts a streamlined architecture with 6 layers and 384-dimensional hidden layers, more efficient compared to full models
WikiText Optimization
Specifically fine-tuned for wikitext data, suitable for processing Wikipedia-style text
Efficient Training
Uses Adam optimizer and linear learning rate scheduler, completing fine-tuning within 3 epochs

Model Capabilities

Text Generation
Language Model Fine-tuning

Use Cases

Text Generation
Wikipedia-style Text Generation
Generates structured text similar to Wikipedia entries
Achieved a loss value of 6.5526 on the wikitext validation set
Education & Research
Language Model Research
Serves as a research benchmark for lightweight language models
Featured Recommended AI Models
AIbase
Empowering the Future, Your AI Solution Knowledge Base
EnglishįŽ€äŊ“中文įšéĢ”中文ãĢãģんご
Š 2025AIbase