diff --git a/.devcontainer/Dockerfile b/.devcontainer/Dockerfile index 979b81bee..7c139ffea 100644 --- a/.devcontainer/Dockerfile +++ b/.devcontainer/Dockerfile @@ -11,7 +11,6 @@ ARG USERNAME=vscode ARG USER_UID=1000 ARG USER_GID=$USER_UID - # Install curl to get ollama # Install build-essential for C++ (needed for ChromaDB) RUN apt-get update && apt-get install -y curl build-essential git git-lfs @@ -19,6 +18,5 @@ RUN apt-get update && apt-get install -y curl build-essential git git-lfs # Install Ollama within Docker container to run large language models locally RUN curl -fsSL https://ollama.com/install.sh | sh - # Switch back to dialog for any ad-hoc use of apt-get ENV DEBIAN_FRONTEND=dialog diff --git a/docs/devcontainer.md b/docs/devcontainer.md index 0b0a3fd28..28f3189b0 100644 --- a/docs/devcontainer.md +++ b/docs/devcontainer.md @@ -1,4 +1,5 @@ --- +diataxis_type: explanation intents: - Provide reader with an overview on how the dev container is built, and what software is installed in dev container. @@ -10,66 +11,226 @@ linked_files: - .devcontainer/Dockerfile - .devcontainer/devcontainer.json --- -# Development Container Overview -The development container for Llamabot is built using a Dockerfile and is influenced by the `devcontainer.json` file. This document provides an overview of how the dev container is built, the software installed within it, and the build steps with examples of possible common failure modes and how to fix them. +# Llamabot Development Container Guide + +This guide provides a comprehensive overview of the Llamabot development container, which ensures a consistent and reproducible development environment. It covers the container's construction, the software it includes, the automated build process, and solutions to common issues. + +## Table of Contents + +1. [Introduction](#introduction) +2. [Building the Development Container](#building-the-development-container) + - [Dockerfile Overview](#dockerfile-overview) + - [Installing Ollama](#installing-ollama) +3. [Devcontainer Configuration](#devcontainer-configuration) + - [Build Context](#build-context) + - [Visual Studio Code Customizations](#visual-studio-code-customizations) + - [Port Forwarding](#port-forwarding) + - [Volume Mounting](#volume-mounting) + - [Post-Create and Post-Start Commands](#post-create-and-post-start-commands) +4. [Automated Build Process](#automated-build-process) + - [GitHub Actions Workflow](#github-actions-workflow) +5. [Common Issues and Solutions](#common-issues-and-solutions) + - [Dockerfile Syntax Errors](#dockerfile-syntax-errors) + - [Missing Dependencies](#missing-dependencies) + - [Package Installation Failures](#package-installation-failures) +6. [Conclusion](#conclusion) + +## Introduction + +To provide a seamless and consistent development experience, Llamabot utilizes a development container. This container ensures that all developers operate within the same environment, eliminating discrepancies caused by differing local setups. It includes all the necessary tools and dependencies required for developing Llamabot. ## Building the Development Container -The development container is built using the Dockerfile located at `.devcontainer/Dockerfile`. The Dockerfile starts with a base image `ghcr.io/prefix-dev/pixi:latest` and sets up the environment by installing necessary software and dependencies. The Dockerfile also adds a non-root user with sudo access and sets up the environment for the development of Llamabot. +The development container is defined by a Dockerfile located at `.devcontainer/Dockerfile`. The build process starts with a base image and progresses by installing additional software and configuring the environment. + +### Dockerfile Overview + +The Dockerfile performs the following key steps: + +1. **Base Image**: Begins with the base image `ghcr.io/prefix-dev/pixi:latest`. +2. **Environment Configuration**: Sets `DEBIAN_FRONTEND=noninteractive` to suppress interactive prompts during package installations. +3. **Software Installation**: + - Updates package lists. + - Installs essential packages such as `curl` and `build-essential`. + - These are necessary for building C++ extensions required by dependencies like ChromaDB. +4. **User Configuration**: + - Creates a non-root user with `sudo` privileges for development purposes. +5. **Ollama Installation**: + - Installs Ollama to enable local execution of large language models within the container. + +An excerpt from the Dockerfile: + +```dockerfile +FROM ghcr.io/prefix-dev/pixi:latest + +ENV DEBIAN_FRONTEND=noninteractive + +# Update package lists and install required packages +RUN apt-get update && apt-get install -y \ + curl \ + build-essential + +# Install Ollama +RUN curl -fsSL https://ollama.com/install.sh | sh + +# Additional user configuration (if applicable) +``` + +### Installing Ollama + +Ollama is crucial for running large language models locally inside the container. It is installed using the following command: + +```dockerfile +RUN curl -fsSL https://ollama.com/install.sh | sh +``` + +This command downloads and executes the Ollama installation script, setting up the environment necessary for Llamabot to function properly. + +## Devcontainer Configuration + +The `devcontainer.json` file, located at `.devcontainer/devcontainer.json`, configures the development container, particularly for use with Visual Studio Code's Remote - Containers extension. + +Key configurations include: + +### Build Context + +Specifies the Dockerfile and context used to build the container: + +```json +"build": { + "dockerfile": "Dockerfile", + "context": ".." +}, +``` + +### Visual Studio Code Customizations + +Defines settings and extensions to enhance the development experience within VS Code: + +```json +"settings": { + // Add VS Code settings here +}, +"extensions": [ + // List of VS Code extensions +], +``` + +### Port Forwarding + +Forwards necessary ports (e.g., port `8888`) to access services running within the container: + +```json +"forwardPorts": [8888], +``` + +### Volume Mounting + +Mounts directories into the container to address filesystem compatibility issues, such as case insensitivity on macOS: + +```json +"mounts": [ + "source=llamabot-pixi,target=/workspace/.pixi,type=volume" +], +``` + +### Post-Create and Post-Start Commands + +Automates environment setup and starts necessary services: + +- **`postCreateCommand`**: Runs after the container is created. It installs dependencies and sets up the development environment. + + ```json + "postCreateCommand": "pixi install && \ + /workspaces/llamabot/.pixi/envs/default/bin/pre-commit install && \ + /workspaces/llamabot/.pixi/envs/default/bin/python -m ipykernel install --user --name llamabot", + ``` + +- **`postStartCommand`**: Runs each time the container starts. It launches the Ollama server or other required services. + + ```json + "postStartCommand": "ollama serve", + ``` + +## Automated Build Process + +To ensure the development container remains up-to-date and accessible, an automated build process is set up using GitHub Actions. + +### GitHub Actions Workflow + +The workflow is defined in `.github/workflows/build-devcontainer.yaml` and is triggered on a schedule or when changes are pushed to the `main` branch. + +Workflow steps include: -### Dockerfile Contents +1. **Set Up QEMU and Docker Buildx**: Configures the environment to support building images for multiple architectures. -The Dockerfile includes the following key steps: + ```yaml + - name: Set up QEMU + uses: docker/setup-qemu-action@v3 -1. pull the base image on an amd platform (we specified the platform at this step because some of the dependencies are not supported on arm) -2. Installs `curl` and `build-essential` for C++ (needed for ChromaDB). -3. Installs Ollama within the Docker container. -4. Sets the final command and switches back to dialog for any ad-hoc use of `apt-get`. + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + ``` -### Devcontainer.json Contents +2. **Authenticate with Docker Hub**: Logs into Docker Hub using stored secrets to allow image pushing. -The `devcontainer.json` file located at `.devcontainer/devcontainer.json` influences the development container by specifying the build context, customizations for Visual Studio Code, forward ports, and post-create and post-start commands. + ```yaml + - name: Login to Docker Hub + uses: docker/login-action@v3 + with: + username: ${{ secrets.DOCKERHUB_USERNAME }} + password: ${{ secrets.DOCKERHUB_TOKEN }} + ``` -1. Specifies the Dockerfile and build context. -2. Customizes Visual Studio Code settings and extensions for the development environment. -3. Forwards port `8888` for the development environment. -4. Mount the `.pixi` directory into a volume. This is needed since the `.pixi` directory shouldn't be on a case insensitive filesystem (default on macOS, Windows) but instead in its own volume. -5. Specifies post-create commands. This includes installing the project within the mounted repo (`/workspaces/llamabot`). installing `pre-commit` hooks and `ipykernel` for setting up the environment and. -6. Specifies post-start commands for running the Llamabot server. +3. **Build and Push the Container**: Builds the development container image and pushes it to Docker Hub with appropriate tags. Uses caching to speed up subsequent builds. -### Purpose of postCreateCommand and postStartCommand + ```yaml + - name: Build and push + uses: docker/build-push-action@v6 + with: + push: true + tags: ericmjl/llamabot-devcontainer:latest, ericmjl/llamabot-devcontainer:${{ github.sha }} + file: .devcontainer/Dockerfile + cache-from: type=gha + cache-to: type=gha,mode=max + ``` -The 'postCreateCommand' is executed after the development container is created to install the pixi project, set up the environment, and the 'postStartCommand' is executed after the container is started to run the Llamabot server. +## Common Issues and Solutions -### Ollama Software +When building or using the development container, you might encounter some common issues. Below are symptoms and resolutions for these problems. -The 'ollama' software is used to run large language models locally within the Docker container and is installed using the command `RUN curl -fsSL https://ollama.com/install.sh | sh`. Ollama is a crucial component for running large language models within the development container. +### Dockerfile Syntax Errors -## Directories within the repo +**Symptoms**: The build process fails with syntax error messages. -### Tests Directory +**Solutions**: -The `tests` directory contains the software tests to get started with development. +- Review the Dockerfile for syntax errors. +- Ensure all commands are correctly formatted. +- Check that all referenced files and directories exist. -### Llamabot Directory +### Missing Dependencies -The 'llamabot' directory contains the source code and documentation for the Llamabot project, highlighting its significance in the development container. +**Symptoms**: Errors indicating missing packages or libraries during the build or at runtime. -## Build Process +**Solutions**: -The build process for the development container is automated using GitHub Actions. The workflow is defined in the `.github/workflows/build-devcontainer.yaml` file. The workflow is triggered on a schedule and on pushes to the main branch. It sets up QEMU, Docker Buildx, and logs in to Docker Hub. It then builds and pushes the development container to Docker Hub. +- Verify that all required packages are listed in the Dockerfile's `apt-get install` commands. +- Check for typos in package names. +- Run `apt-get update` before installing packages to ensure the package lists are up-to-date. -### Build Process Workflow +### Package Installation Failures -1. Sets up QEMU and Docker Buildx. -2. Logs in to Docker Hub using secrets. -3. Builds and pushes the development container to Docker Hub with appropriate tags and caching configurations. +**Symptoms**: Failure messages during `apt-get install` or other package installations. -## Common Failure Modes +**Solutions**: -Common failure modes in the build process may include issues with Dockerfile syntax, missing dependencies, or failed package installations. These issues can be resolved by carefully reviewing the Dockerfile, ensuring all necessary files are copied, and troubleshooting package installations, including the installation process for the 'ollama' software. +- Check network connectivity from the build environment. +- Ensure package repositories are reachable and not experiencing downtime. +- Examine the logs for specific error messages to identify the cause. +- Retry the build; transient network issues can sometimes cause failures. ## Conclusion -This updated documentation provides an overview of the development container for Llamabot, including the build process, influence of the devcontainer.json file, and common failure modes in the build process. Developers can use this documentation to understand how the development container is built and how to troubleshoot common issues during the build process. +By utilizing a development container, Llamabot ensures a consistent and efficient development environment for all contributors. This guide has detailed the setup and configuration of the development container, the automated build process, and troubleshooting steps for common issues. Following these guidelines will facilitate a smooth development workflow and help maintain consistency across development environments. diff --git a/tests/cli/test_docs.py b/tests/cli/test_docs.py index f99ad8820..5679a25f4 100644 --- a/tests/cli/test_docs.py +++ b/tests/cli/test_docs.py @@ -13,8 +13,12 @@ DocsContainFactuallyIncorrectMaterial, SourceContainsContentNotCoveredInDocs, DocsDoNotCoverIntendedMaterial, + DiataxisType, + diataxis_sources, ) from pyprojroot import here +from unittest.mock import patch + ### Evals system_prompt1 = """You are an expert in documentation management. @@ -225,3 +229,31 @@ def test_out_of_date_when_source_changes( assert doc_issue.status == expected_status if doc_issue.status: assert doc_issue.reasons + + +def test_markdown_source_file_diataxis(): + """Test that the MarkdownSourceFile can correctly identify the diataxis type and source.""" + # Create a temporary markdown file with diataxis type + test_content = """--- +diataxis_type: howto +--- +# Test Content +""" + with patch("requests.get") as mock_get: + # Mock the response from the diataxis source + mock_get.return_value.status_code = 200 + mock_get.return_value.text = "Mock diataxis guide content" + + with pytest.tmp_path() as tmp_path: + test_file = tmp_path / "test.md" + test_file.write_text(test_content) + + # Create MarkdownSourceFile instance + md_file = MarkdownSourceFile(test_file) + + # Check if diataxis type and source are properly set + assert md_file.diataxis_type == DiataxisType.HOWTO + assert md_file.diataxis_source == "Mock diataxis guide content" + + # Check if the request was made with the correct URL + mock_get.assert_called_once_with(diataxis_sources[DiataxisType.HOWTO])