Text Generation WebUI

Overview

Text Generation WebUI is an open-source Gradio-based web interface for running Large Language Models (LLMs) locally. Created by oobabooga, it supports multiple text generation backends including Transformers, llama.cpp, and ExLlamaV2. The interface provides multiple chat modes, model fine-tuning capabilities, and an OpenAI-compatible API for seamless integration with other applications.

System Requirements

• CPU: 4+ cores (8+ cores recommended for larger models)
• RAM: 16GB+ (32GB+ recommended for larger models)
• GPU:
  • NVIDIA GPU with 8GB+ VRAM for smaller models (7B parameters)
  • NVIDIA GPU with 24GB+ VRAM for larger models (up to 13B parameters)
  • Multiple GPUs or high-end GPUs for very large models (30B+ parameters)
• Storage: 20GB+ (100GB+ recommended for storing multiple models)

Installation Guide

Prerequisites

• Python 3.8 or higher
• Git
• NVIDIA CUDA drivers if using an NVIDIA GPU
• Conda/Miniconda (recommended for environment management)

Option 1: One-Click Installers (Simplest)

For Windows, macOS, and Linux users, oobabooga offers one-click installers that automate the entire setup process:

1. Download the appropriate installer for your system from: One-Click Installers Repository

2. Run the installer and follow the on-screen instructions.

3. Once installed, the interface will be accessible at: http://localhost:7860

Option 2: Manual Installation

1. Clone the repository:

   git clone https://github.com/oobabooga/text-generation-webui
   cd text-generation-webui
   

2. Create and activate a conda environment:

   conda create -n textgen python=3.10
   conda activate textgen
   

3. Install the dependencies based on your system:

   # For NVIDIA GPUs
   pip install -r requirements.txt
   # For CPU-only systems
   pip install -r requirements_cpu_only.txt
   

4. Start the web UI:

   # Basic startup
   python server.py
   
   # Or with additional parameters
   python server.py --listen --chat
   

Option 3: Docker Installation

1. Pull the Docker image:

   docker pull atinoda/text-generation-webui:default-nvidia
   

2. Run the container:

   docker run -it --rm -e EXTRA_LAUNCH_ARGS="--listen --verbose" --gpus all -p 7860:7860 atinoda/text-generation-webui:default-nvidia
   

Practical Exercise: Running Your First LLM

Let’s set up and run a language model with Text Generation WebUI.

Step 1: Download a Model

After installing Text Generation WebUI, you’ll need to download a model:

1. Start the web UI:

   python server.py
   

2. Open your browser and navigate to http://localhost:7860

3. Go to the “Model” tab

4. Select a model repository like “TheBloke” and choose a model. For beginners, try one of these smaller models:

   • TheBloke/Llama-2-7B-Chat-GGUF
   • TheBloke/Mistral-7B-Instruct-v0.2-GGUF

5. Select a quantization level based on your GPU VRAM (4-bit for lower VRAM, 8-bit for more VRAM)

6. Click “Download” and wait for the model to download

Step 2: Load the Model

1. Once downloaded, select your model from the “Model” dropdown

2. Set appropriate parameters:

   • Context length: 2048 (or higher if your model supports it)
   • n_gpu_layers: Set to -1 to offload all layers to GPU
   • n_batch: 512 (reduce if you encounter memory issues)

3. Click “Load” to load the model into memory

Step 3: Chat with the Model

1. Navigate to the “Chat” tab

2. Select “Chat-Instruct” mode from the dropdown for instruction-following models

3. Start a conversation by typing a message and pressing Enter

4. Experiment with different parameters in the “Generation” section to control output:

   • Temperature: Controls randomness (lower for more deterministic responses)
   • Top P: Controls diversity of responses
   • Max new tokens: Controls maximum response length

Step 4: Exploring Advanced Features

Once you’re comfortable with the basics, try these advanced features:

1. Character Creation:

   • Create custom characters with specific personalities
   • Define custom chat templates and formats

2. Extensions:

   • Enable extensions like API, Memory, or Image Generation from the “Extensions” tab

3. Notebook Mode:

   • Use the “Default” tab for free-form text generation without chat context

4. API Integration:

   • Enable the API extension to integrate with other applications
   • Use the OpenAI-compatible endpoint at http://localhost:5000/v1

Resources

Official Documentation

The GitHub repository contains detailed instructions and documentation:

GitHub Repository

Community Support

Join the community to get help, share experiences, and contribute to the project:

GitHub Issues

Extensions

Explore available extensions to enhance functionality:

Text Generation WebUI Extensions

Tutorials and Guides

Community tutorials for various aspects of the project:

GitHub Discussions