How to persist container logs to disk

[lanedsmu]

The single-container docker image for Lowcoder includes logging to disk and default volume mapping for accessing those logs on the Docker host. The multi-container deployment doesn't do this, however, falling back on Docker's default, which is to log all container output to stdout (or stderr), and surface that output through the docker logs command. Logging to persistent storage is especially important in a production environment, so here is one way to configure your Docker environment to support this.

Docker provides several mechanisms for handling logs ( see Docker logging documentation here ). The json-file driver would at first blush appear to be a straightforward choice, however there are a couple of potential problems that lead us to another path. The first is that the file location for these logs is not configurable: they are saved in /var/lib/docker/containers/<container_id>/<container_id>-json.log, and this mount point isn't typically sized or equipped for production logging. The second problem is that the Docker daemon doesn't support interacting with these logs files while the container is running, which limits their functionality to post-hoc analysis.

Instead of the json-file driver, we'll look at how to configure the syslog driver. This has some advantages, as most server systems already are running a syslog daemon, and there are a variety of tools in use across enterprises to parse, analyze, and monitor syslog messages.

Configure the Docker Daemon

System-wide configuration

A system-wide configuration is the simplest--but not only--option. The Docker daemon is controlled using the daemon.json file at /etc/docker. We'll add the following

{
  "log-driver":  "syslog",
  "log-opts":  {
    "syslog-facility": "local6",
    "tag":  "{{.Name}}:{{.ID}}"
  },
<other settings>
}

Apart from choosing the relevant log-driver (syslog), we're also adding sonme tags. The "tag" line addes a prefix of "service name:containerID" to each log line. This can help significantly in identifying which log conmes from which app.

Likewise, the "local6" facility is a Linux syslog identifier for local apps that don't fall into one of several other categories. Basically, you can use "local[0-7]" as you choose. See this page for details.

After making this change, we need to restart the Docker daemon, and how to do this depends on the OS you're running. Enterprise Linux distributions based on Red Hat typically will use systemctl:
sudo systemctl restart docker
Ubuntu-based systems will typically use service:
sudo service docker restart

Once this change is made, logs--stdout and stderr--from all Docker containers on the host will be saved to the syslog destination. By default, this typically means they're saved to /var/log/messages.

Individual container configuration

If you're unable to access system-wide Docker configuration, you can use the same log-driver setting in your docker-compose.yml file. The formatting is somewhat different, but the values remain the same:

version: "3"
services:

  ##
  ## Start Lowcoder backend services (api-service and node-service)
  ##
  lowcoder-api-service:
    image: lowcoder-ce-api-service:latest
    container_name: lowcoder-api-service
    # Enabled ports to be able to access backend from host
    # ports:
    #   - "8080:8080"
    logging:
      driver: syslog
      options:
        syslog-facility: local6
        tag: "{{.Name}}:{{.ID}}"
   <rest of the yml file>

The logging section is specific to the service in which it resides, so you'll need to add that section to each of the services for which you want to persist logs.