Setting up OpenLLM with Ollama Web UI

This guide will walk you through setting up OpenLLM with Ollama Web UI on your local machine or any server. The setup involves several steps, which we’ll go through in detail:

Prerequisites
- Windows 11 or Ubuntu 22.04 PC with at least 8GB of RAM (16GB recommended)
- Nvidia GPU with Official Driver Installed (Optional, but recommended)
Docker Installation (Linux/WSL)
Activate GPU compatibility
Install OpenLLM and Ollama Web UI
Run it
Optional: Activate WSL Server Mode
Accessing the Ollama Web UI
Shutting down your container

Let’s get started with each step:

Prerequisites: To start with, make sure you have a Windows 11 or Ubuntu 22.04 PC (or any other Linux distribution) with at least 8GB of RAM. If you want to use a GPU for faster model training and inference, make sure your machine has an Nvidia GPU with the official driver installed.
Docker Installation: To install Docker on Windows/WSL, follow the instructions provided in the official documentation. For Linux systems, you can find installation steps on Docker’s website. Once Docker is installed, make sure to add the current user to the docker group and restart the service.
Activate GPU compatibility: If you want to use a GPU for faster model training and inference, you need to activate GPU compatibility. You can do this by following the instructions provided in the official documentation.
Install OpenLLM and Ollama Web UI:
- Open a terminal or command prompt and navigate to the directory where you want to install OpenLLM and Ollama Web UI. For example, you can create a new directory named ollama using the following command:
```
mkdir $HOME/ollama && cd $HOME/ollama
```
- Install OpenLLM by running the following command:
```
docker run --rm -v $(pwd)/ollama:/root/.ollama ollama/ollama:latest bash -c 'ollama_train --model=13b_v1.2 --dataset=https://huggingface.co/datasets/Lmsysqa/test'
```
- Install Ollama Web UI by running the following command:
```
docker run --rm -p 8080:8080 -v $(pwd)/ollama-webui:/app/backend/data ghcr.io/ollama-webui/ollama-webui:main bash -c 'pip install uvicorn && uvicorn main:app'
```
- After installing OpenLLM and Ollama Web UI, you can start the containers by running the following command:
```
docker run --rm -p 11434:11434 ollama/ollama:latest bash -c 'ollama_train --model=13b_v1.2 --dataset=https://huggingface.co/datasets/Lmsysqa/test'
```
- After starting the containers, you can access the Ollama Web UI by opening a web browser and navigating to http://localhost:8080.
Run it:
- To start the containers, you can run the following command:
```
docker run --rm -p 11434:11434 ollama/ollama:latest bash -c 'ollama_train --model=13b_v1.2 --dataset=https://huggingface.co/datasets/Lmsysqa/test'
```
- After starting the containers, you can access the Ollama Web UI by opening a web browser and navigating to http://localhost:8080.
Optional: Activate WSL Server Mode: If you are using Windows Subsystem for Linux (WSL), you need to activate WSL server mode before running the containers. You can do this by following the instructions provided in the official documentation.
Accessing the Ollama Web UI:
- To access the Ollama Web UI, you can open a web browser and navigate to http://localhost:8080. Once you have accessed the Ollama Web UI, you can start interacting with OpenLLM by using your web browser.
Shutting down your container:
- To shut down your containers, you can run the following command:
```
docker stop $(docker ps --format "{{.ID}}")
```

Extras:

To use Docker Compose files to build and run your application, you can follow these steps:

Create a new directory for your project (e.g., “my-project”) and navigate into it.
Create a file named docker-compose.yml in the project directory with the following contents:

version: '3.8'
services:

  ollama-webui:
    image: ghcr.io/ollama-webui/ollama-webui:main
    container_name: ollama-webui
    volumes:
       - $HOME/ollama/ollama-webui:/app/backend/data
    ports:
       - 3000:8080
    extra_hosts:
       - host.docker.internal:host-gateway
    restart: unless-stopped

  ollama:
    volumes:
       -  $HOME/ollama/ollama:/root/.ollama
    container_name: ollama
    pull_policy: always
    tty: true
    restart: unless-stopped
    image: ollama/ollama:latest
    ports:
       - 11434:11434
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

  ollama-webui:
    image: ghcr.io/ollama-webui/ollama-webui:main
    container_name: ollama-webui
    volumes:
       - $HOME/ollama/ollama-webui:/app/backend/data
    depends_on:
       - ollama
    ports:
       - 3000:8080
    environment:
       - '/ollama/api=http://ollama:11434/api'
    extra_hosts:
       - host.docker.internal:host-gateway
    restart: unless-stopped

Save the file and navigate back to your project directory in the terminal.
Run docker compose up to start the application. Docker will automatically build the images using the Dockerfiles found in the current directory, and then run the containers.

This command will create two containers for your application: “ollama-webui” and “ollama”. The containers will be mapped to ports 3000 and 11434 on your machine when running. You can access the web interface by visiting http://localhost:3000 in your web browser, and the Ollama API endpoint will be available at http://localhost:11434/api.

The docker-compose.yml file specifies the configuration needed to build and run your application using Docker Compose. It defines three services: “ollama-webui”, “ollama”, and “ollama-webui” (with a typo). The services will be mapped to ports 3000, 11434, and 3000 on your machine when running.

After following these steps, you can use Docker Compose commands to manage and interact with your application. For example, to stop the containers, run docker compose down.

That’s it! You have successfully set up OpenLLM with Ollama Web UI on your local machine or any server. Now you can start interacting with OpenLLM by using your web browser.