How to get logs from a Kubernetes charm

See also: How to debug a charm

This document shows how to get logs from a Kubernetes charm.

Contents:

• Use juju debug-log
• Use kubectl logs
• Use pebble logs
• Use pebble changes and tasks
• Use juju ssh and cat
• Use an integrated solution

Use juju debug-log

Juju automatically logs all the agents on a per-model basic. It also picks up automatically logs from charm code that uses the python logging facility:

import logging
logger = logging.getLogger(__name__)

One way to get logs from a Kubernetes charm is thus to use Juju’s built-in logging tools, as follows:

1. Set the model configuration such that logging for a unit is at the DEBUG level:

juju model-config logging-config="<root>=WARNING; unit=DEBUG"

2. View the resulting logs:

juju debug-log

See more: Juju | How to manage Juju logs

Use kubectl logs

With kubectl logs we can see logs from the charm and the workload containers.

For example:

# Get logs from the charm container
kubectl -n model_name logs pods/prometheus-0 -c charm

# Get logs from the workload container
kubectl -n model_name logs pods/prometheus-0 -c prometheus

Logs that are printed to stdout/stderr are automatically picked up by kubectl (in machine charms: journalctl). Logging to stdout/stderr is an established convention and is generally preferable to only logging to a file.

Use pebble logs

With pebble logs we can get logs for a particular pebble service:

juju ssh --container prometheus prometheus/0 \
  /charm/bin/pebble logs prometheus

Use pebble changes and tasks

To see logs related to a service that failed to start, you first obtain the ID of the failed change from the list of changes with pebble changes,

$ juju ssh --container prometheus prometheus/0 /charm/bin/pebble changes
ID   Status  Spawn                   Ready                   Summary
30   Error   yesterday at 21:31 UTC  yesterday at 21:31 UTC  Replan service "prometheus"
31   Done    yesterday at 21:38 UTC  yesterday at 21:38 UTC  Execute command "/usr/bin/promtool"
32   Done    yesterday at 21:38 UTC  yesterday at 21:38 UTC  Replan service "prometheus"

and then query for the logs with pebble tasks,

$ juju ssh --container prometheus prometheus/0 /charm/bin/pebble tasks 30
Status  Spawn                   Ready                   Summary
Error   yesterday at 21:31 UTC  yesterday at 21:31 UTC  Start service "prometheus"

......................................................................
Start service "prometheus"

2023-03-07T21:31:39Z INFO Most recent service output:
    (...)
    ts=2023-03-07T21:31:39.309Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090
    ts=2023-03-07T21:31:39.309Z caller=main.go:807 level=error msg="Unable to start web listener" err="listen tcp 0.0.0.0:9090: bind: address already in use"
2023-03-07T21:31:39Z ERROR cannot start service: exited quickly with code 1

Use juju ssh and cat

You could cat log files directly with juju ssh:

juju ssh --container prometheus prometheus/0 \
  cat /var/log/bootstrap.log

If your workload does not write logs to disk, you could modify the pebble service as follows:

"services": {
  "service-name": {
    # trick to drop the logs to a file but also keep them available in the pod logs
    # https://github.com/canonical/traefik-k8s-operator/blob/a287fa6a41077c8c3b7d3eb244f055c2a354bd2a/src/charm.py#L880
    "command": '/bin/sh -c "{} | tee -a {}"'.format(BIN_PATH, LOG_PATH),
  },
},

Use an integrated solution

For an integrated solution, consider the loki_push_api interface.