Skip to content

Commit

Permalink
docs(devcontainer)📝: Update development container documentation and D…
Browse files Browse the repository at this point in the history 
…ockerfile

- Revise and expand the development container guide in the documentation.
- Remove unnecessary whitespace in Dockerfile.
- Add diataxis type to devcontainer documentation.
  • Loading branch information
@ericmjl
ericmjl committed Nov 29, 2024
1 parent 687ff89 commit f34e12e
Show file tree
Hide file tree
Showing 3 changed files with 228 additions and 37 deletions.
2 changes: 0 additions & 2 deletions .devcontainer/Dockerfile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,12 @@ ARG USERNAME=vscode
ARG USER_UID=1000
ARG USER_GID=$USER_UID


# Install curl to get ollama
# Install build-essential for C++ (needed for ChromaDB)
RUN apt-get update && apt-get install -y curl build-essential git git-lfs

# Install Ollama within Docker container to run large language models locally
RUN curl -fsSL https://ollama.com/install.sh | sh


# Switch back to dialog for any ad-hoc use of apt-get
ENV DEBIAN_FRONTEND=dialog
231 changes: 196 additions & 35 deletions docs/devcontainer.md
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
---
diataxis_type: explanation
intents:
- Provide reader with an overview on how the dev container is built, and what software
is installed in dev container.
Expand All @@ -10,66 +11,226 @@ linked_files:
- .devcontainer/Dockerfile
- .devcontainer/devcontainer.json
---
# Development Container Overview

The development container for Llamabot is built using a Dockerfile and is influenced by the `devcontainer.json` file. This document provides an overview of how the dev container is built, the software installed within it, and the build steps with examples of possible common failure modes and how to fix them.
# Llamabot Development Container Guide

This guide provides a comprehensive overview of the Llamabot development container, which ensures a consistent and reproducible development environment. It covers the container's construction, the software it includes, the automated build process, and solutions to common issues.

## Table of Contents

1. [Introduction](#introduction)
2. [Building the Development Container](#building-the-development-container)
- [Dockerfile Overview](#dockerfile-overview)
- [Installing Ollama](#installing-ollama)
3. [Devcontainer Configuration](#devcontainer-configuration)
- [Build Context](#build-context)
- [Visual Studio Code Customizations](#visual-studio-code-customizations)
- [Port Forwarding](#port-forwarding)
- [Volume Mounting](#volume-mounting)
- [Post-Create and Post-Start Commands](#post-create-and-post-start-commands)
4. [Automated Build Process](#automated-build-process)
- [GitHub Actions Workflow](#github-actions-workflow)
5. [Common Issues and Solutions](#common-issues-and-solutions)
- [Dockerfile Syntax Errors](#dockerfile-syntax-errors)
- [Missing Dependencies](#missing-dependencies)
- [Package Installation Failures](#package-installation-failures)
6. [Conclusion](#conclusion)

## Introduction

To provide a seamless and consistent development experience, Llamabot utilizes a development container. This container ensures that all developers operate within the same environment, eliminating discrepancies caused by differing local setups. It includes all the necessary tools and dependencies required for developing Llamabot.

## Building the Development Container

The development container is built using the Dockerfile located at `.devcontainer/Dockerfile`. The Dockerfile starts with a base image `ghcr.io/prefix-dev/pixi:latest` and sets up the environment by installing necessary software and dependencies. The Dockerfile also adds a non-root user with sudo access and sets up the environment for the development of Llamabot.
The development container is defined by a Dockerfile located at `.devcontainer/Dockerfile`. The build process starts with a base image and progresses by installing additional software and configuring the environment.

### Dockerfile Overview

The Dockerfile performs the following key steps:

1. **Base Image**: Begins with the base image `ghcr.io/prefix-dev/pixi:latest`.
2. **Environment Configuration**: Sets `DEBIAN_FRONTEND=noninteractive` to suppress interactive prompts during package installations.
3. **Software Installation**:
- Updates package lists.
- Installs essential packages such as `curl` and `build-essential`.
- These are necessary for building C++ extensions required by dependencies like ChromaDB.
4. **User Configuration**:
- Creates a non-root user with `sudo` privileges for development purposes.
5. **Ollama Installation**:
- Installs Ollama to enable local execution of large language models within the container.

An excerpt from the Dockerfile:

```dockerfile
FROM ghcr.io/prefix-dev/pixi:latest

ENV DEBIAN_FRONTEND=noninteractive

# Update package lists and install required packages
RUN apt-get update && apt-get install -y \
curl \
build-essential

# Install Ollama
RUN curl -fsSL https://ollama.com/install.sh | sh

# Additional user configuration (if applicable)
```

### Installing Ollama

Ollama is crucial for running large language models locally inside the container. It is installed using the following command:

```dockerfile
RUN curl -fsSL https://ollama.com/install.sh | sh
```

This command downloads and executes the Ollama installation script, setting up the environment necessary for Llamabot to function properly.

## Devcontainer Configuration

The `devcontainer.json` file, located at `.devcontainer/devcontainer.json`, configures the development container, particularly for use with Visual Studio Code's Remote - Containers extension.

Key configurations include:

### Build Context

Specifies the Dockerfile and context used to build the container:

```json
"build": {
"dockerfile": "Dockerfile",
"context": ".."
},
```

### Visual Studio Code Customizations

Defines settings and extensions to enhance the development experience within VS Code:

```json
"settings": {
// Add VS Code settings here
},
"extensions": [
// List of VS Code extensions
],
```

### Port Forwarding

Forwards necessary ports (e.g., port `8888`) to access services running within the container:

```json
"forwardPorts": [8888],
```

### Volume Mounting

Mounts directories into the container to address filesystem compatibility issues, such as case insensitivity on macOS:

```json
"mounts": [
"source=llamabot-pixi,target=/workspace/.pixi,type=volume"
],
```

### Post-Create and Post-Start Commands

Automates environment setup and starts necessary services:

- **`postCreateCommand`**: Runs after the container is created. It installs dependencies and sets up the development environment.

```json
"postCreateCommand": "pixi install && \
/workspaces/llamabot/.pixi/envs/default/bin/pre-commit install && \
/workspaces/llamabot/.pixi/envs/default/bin/python -m ipykernel install --user --name llamabot",
```

- **`postStartCommand`**: Runs each time the container starts. It launches the Ollama server or other required services.

```json
"postStartCommand": "ollama serve",
```

## Automated Build Process

To ensure the development container remains up-to-date and accessible, an automated build process is set up using GitHub Actions.

### GitHub Actions Workflow

The workflow is defined in `.github/workflows/build-devcontainer.yaml` and is triggered on a schedule or when changes are pushed to the `main` branch.

Workflow steps include:

### Dockerfile Contents
1. **Set Up QEMU and Docker Buildx**: Configures the environment to support building images for multiple architectures.

The Dockerfile includes the following key steps:
```yaml
- name: Set up QEMU
uses: docker/setup-qemu-action@v3

1. pull the base image on an amd platform (we specified the platform at this step because some of the dependencies are not supported on arm)
2. Installs `curl` and `build-essential` for C++ (needed for ChromaDB).
3. Installs Ollama within the Docker container.
4. Sets the final command and switches back to dialog for any ad-hoc use of `apt-get`.
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
```
### Devcontainer.json Contents
2. **Authenticate with Docker Hub**: Logs into Docker Hub using stored secrets to allow image pushing.
The `devcontainer.json` file located at `.devcontainer/devcontainer.json` influences the development container by specifying the build context, customizations for Visual Studio Code, forward ports, and post-create and post-start commands.
```yaml
- name: Login to Docker Hub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
```
1. Specifies the Dockerfile and build context.
2. Customizes Visual Studio Code settings and extensions for the development environment.
3. Forwards port `8888` for the development environment.
4. Mount the `.pixi` directory into a volume. This is needed since the `.pixi` directory shouldn't be on a case insensitive filesystem (default on macOS, Windows) but instead in its own volume.
5. Specifies post-create commands. This includes installing the project within the mounted repo (`/workspaces/llamabot`). installing `pre-commit` hooks and `ipykernel` for setting up the environment and.
6. Specifies post-start commands for running the Llamabot server.
3. **Build and Push the Container**: Builds the development container image and pushes it to Docker Hub with appropriate tags. Uses caching to speed up subsequent builds.
### Purpose of postCreateCommand and postStartCommand
```yaml
- name: Build and push
uses: docker/build-push-action@v6
with:
push: true
tags: ericmjl/llamabot-devcontainer:latest, ericmjl/llamabot-devcontainer:${{ github.sha }}
file: .devcontainer/Dockerfile
cache-from: type=gha
cache-to: type=gha,mode=max
```
The 'postCreateCommand' is executed after the development container is created to install the pixi project, set up the environment, and the 'postStartCommand' is executed after the container is started to run the Llamabot server.
## Common Issues and Solutions
### Ollama Software
When building or using the development container, you might encounter some common issues. Below are symptoms and resolutions for these problems.
The 'ollama' software is used to run large language models locally within the Docker container and is installed using the command `RUN curl -fsSL https://ollama.com/install.sh | sh`. Ollama is a crucial component for running large language models within the development container.
### Dockerfile Syntax Errors
## Directories within the repo
**Symptoms**: The build process fails with syntax error messages.
### Tests Directory
**Solutions**:
The `tests` directory contains the software tests to get started with development.
- Review the Dockerfile for syntax errors.
- Ensure all commands are correctly formatted.
- Check that all referenced files and directories exist.
### Llamabot Directory
### Missing Dependencies
The 'llamabot' directory contains the source code and documentation for the Llamabot project, highlighting its significance in the development container.
**Symptoms**: Errors indicating missing packages or libraries during the build or at runtime.
## Build Process
**Solutions**:
The build process for the development container is automated using GitHub Actions. The workflow is defined in the `.github/workflows/build-devcontainer.yaml` file. The workflow is triggered on a schedule and on pushes to the main branch. It sets up QEMU, Docker Buildx, and logs in to Docker Hub. It then builds and pushes the development container to Docker Hub.
- Verify that all required packages are listed in the Dockerfile's `apt-get install` commands.
- Check for typos in package names.
- Run `apt-get update` before installing packages to ensure the package lists are up-to-date.

### Build Process Workflow
### Package Installation Failures

1. Sets up QEMU and Docker Buildx.
2. Logs in to Docker Hub using secrets.
3. Builds and pushes the development container to Docker Hub with appropriate tags and caching configurations.
**Symptoms**: Failure messages during `apt-get install` or other package installations.

## Common Failure Modes
**Solutions**:

Common failure modes in the build process may include issues with Dockerfile syntax, missing dependencies, or failed package installations. These issues can be resolved by carefully reviewing the Dockerfile, ensuring all necessary files are copied, and troubleshooting package installations, including the installation process for the 'ollama' software.
- Check network connectivity from the build environment.
- Ensure package repositories are reachable and not experiencing downtime.
- Examine the logs for specific error messages to identify the cause.
- Retry the build; transient network issues can sometimes cause failures.

## Conclusion

This updated documentation provides an overview of the development container for Llamabot, including the build process, influence of the devcontainer.json file, and common failure modes in the build process. Developers can use this documentation to understand how the development container is built and how to troubleshoot common issues during the build process.
By utilizing a development container, Llamabot ensures a consistent and efficient development environment for all contributors. This guide has detailed the setup and configuration of the development container, the automated build process, and troubleshooting steps for common issues. Following these guidelines will facilitate a smooth development workflow and help maintain consistency across development environments.
32 changes: 32 additions & 0 deletions tests/cli/test_docs.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,12 @@
DocsContainFactuallyIncorrectMaterial,
SourceContainsContentNotCoveredInDocs,
DocsDoNotCoverIntendedMaterial,
DiataxisType,
diataxis_sources,
)
from pyprojroot import here
from unittest.mock import patch


### Evals
system_prompt1 = """You are an expert in documentation management.
Expand Down Expand Up @@ -225,3 +229,31 @@ def test_out_of_date_when_source_changes(
assert doc_issue.status == expected_status
if doc_issue.status:
assert doc_issue.reasons


def test_markdown_source_file_diataxis():
"""Test that the MarkdownSourceFile can correctly identify the diataxis type and source."""
# Create a temporary markdown file with diataxis type
test_content = """---
diataxis_type: howto
---
# Test Content
"""
with patch("requests.get") as mock_get:
# Mock the response from the diataxis source
mock_get.return_value.status_code = 200
mock_get.return_value.text = "Mock diataxis guide content"

with pytest.tmp_path() as tmp_path:
test_file = tmp_path / "test.md"
test_file.write_text(test_content)

# Create MarkdownSourceFile instance
md_file = MarkdownSourceFile(test_file)

# Check if diataxis type and source are properly set
assert md_file.diataxis_type == DiataxisType.HOWTO
assert md_file.diataxis_source == "Mock diataxis guide content"

# Check if the request was made with the correct URL
mock_get.assert_called_once_with(diataxis_sources[DiataxisType.HOWTO])

0 comments on commit f34e12e

Please sign in to comment.