How to manage agent logs

See also: Log

Juju saves or discards logs according to the value of the model config key logging-config. Therefore, logging-config needs to be set before the events you want to collect logs for (i.e. before attempting to reproduce a bug).

Set values.

To change the logging configuration for machine and unit agents:
Run the model-config command with the logging-config key set to a "-enclosed, semi-colon-separated list of <filter>=<verbosity level> pairs.

Example setting machine agent logs to `WARNING` and unit agent logs to `TRACE`

juju model-config logging-config="<root>=WARNING;unit=TRACE"

Example setting unit agent logs for unit `0` of `mysql` to `DEBUG`

juju model-config logging-config="unit.mysql/0=DEBUG"

See more: How to configure a model, List of model configuration keys > logging-config

To avoid filling up the database unnecessarily:
When verbose logging is no longer needed, return logging to normal levels!

To change the logging configuration on a per-unit-agent basis:

SSH into the unit’s machine. E.g., for mysql/0:

juju ssh mysql/0

Open the unit’s agent configuration file. For our example, it will be /var/lib/juju/agents/unit-mysql-0/agent.conf/ Then, find the values section, and add a line with the field LOGGING_OVERRIDE set to juju=<desired log level>, below TRACE. The bottom of the file should now look as below:

loggingconfig: <root>=WARNING;unit=DEBUG
values:
  CONTAINER_TYPE: ""
  NAMESPACE: ""
  LOGGING_OVERRIDE: juju=trace
mongoversion: "0.0"

Restart the affected agent:

sudo systemctl restart jujud-unit-mysql-0.service

Get values. To verify the current logging configuration for machine and unit agents, run model-config followed by the logging-config key:

juju model-config logging-config

Sample output:

<root>=WARNING;unit=DEBUG;#http=TRACE

which means that the machine agent (<root>) log level is set to WARNING, the unit agent (unit) log level is set at DEBUG, and the http label is set to TRACE.

See more: How to configure a model, List of model configuration keys > logging-config

Forward logs to an external logsink

You can optionally forward log messages to a remote syslog server over a secure TLS connection, on a per-model basis, as below:

See Rsyslog documentation for help with security-related files (certificates, keys) and the configuration of the remote syslog server.

Configure the controller for remote logging by configuring it during controller creation as below:

juju bootstrap <cloud> --config mylogconfig.yaml

where the YAML file is as below:

syslog-host: <host>:<port>
syslog-ca-cert: |
-----BEGIN CERTIFICATE-----
 <cert-contents>
-----END CERTIFICATE-----
syslog-client-cert: |
-----BEGIN CERTIFICATE-----
 <cert-contents>
-----END CERTIFICATE-----
syslog-client-key: |
-----BEGIN PRIVATE KEY-----
 <cert-contents>
-----END PRIVATE KEY-----

See more: How to configure a controller

Enable log forwarding for a model by configuring it as below:

juju model-config -m <model> logforward-enabled=True

An initial 100 (maximum) existing log lines will be forwarded.

See more: How to configure a model

Pro tip: You can configure remote logging and enable log forwarding on all the controller’s models in one step by running

juju bootstrap <cloud> --config mylogconfig.yaml --config logforward-enabled=True

Manage the log files

Only applicable for machines – for Kubernetes logs are written directly to stdout of the container and can be retrieved with native Kubernetes methods, e.g., kubectl logs -c <container-name> <pod-name> -n <model-name> .

See more: Juju agent logs – machines, Juju agent logs – Kubernetes

View the log files

To view the Juju log files in a Juju machine:

Open a shell into the machine:

(1a) IfJuju can connect to the machine (i.e., the output contains Connected to <IP address>) and the machine is fully provisioned, use juju ssh. For example, to connect to machine 0:

juju ssh 0

(1b) IfJuju can connect to the machine (i.e., the output contains Connected to <IP address>) but the machine is not fully provisioned (e.g., command hangs at Running machine configuration script...), use the ssh command followed by the address of the machine and the path to the place where Juju stores your SSH keys (including the ones it generates automatically for you):

ssh ubuntu@<ip-address> -i <juju-data-dir>/ssh/juju_id_rsa

Here, <juju-data-dir> defaults to ~/.local/share/juju, but if you’ve set the JUJU_DATA environment variable, it will be equal to that instead.

(1c) If Juju cannot connect to the machine (i.e., the command never reaches Connected to <IP address>), use cloud-specific tools. For example, for the LXD cloud:


lxc exec [container-name] bash

or, for the MicroK8s cloud:

microk8s kubectl exec controller-0 -itc api-server -n [namespace] -- bash

See more: How to access a machine via SSH

Examine the log files under /var/log with commands such as cat, less, or tail -f, for example:

cat /var/log/juju

Which log to look at depends on the type of failure, but generally speaking, syslog, cloud-init.log, cloud-init-output.log, and /var/log/juju are good ones to look at.

Expand to see an example for a controller machine

# SSH into machine 0 of the controller model:
juju ssh -m controller 0

# Navigate to the logs directory:
cd /var/log/juju

# List the contents:
ls

# View, e.g., the audit log file:
cat audit.log

Control the log file rotation

See also: List of controller configuration keys

Juju has settings to control the rotation of the various log files it produces.

There are two settings for each log file type: maximum size and number of backups. When the current log file of a particular type reaches the maximum size, Juju renames the log file to include a timestamp and gzips it, producing a “backup” log file.

Here’s an example of the controller’s machine agent logs with the maximum size set to 1MB, showing two timestamped backups as well as the current log file:

$ juju bootstrap localhost --config agent-logfile-max-size=1MB
$ lxc exec juju-6bf629-0 -- ls -l /var/log/juju
...
-rw-r----- 1 syslog adm   3577 Jan 12 02:01 machine-0-2022-01-12T02-01-07.995.log.gz
-rw-r----- 1 syslog adm   3578 Jan 12 02:02 machine-0-2022-01-12T02-02-08.011.log.gz
-rw-r----- 1 syslog adm 600000 Jan 12 02:02 machine-0.log

The full list of the controller settings that configure log file rotation is shown below. Normally these are set at bootstrap time with the --config option (see How to configure a controller).

The following config settings configure agent log files, including the API server “log sink”, the machine agent logs on controller and unit machines, and the unit agent logs:

agent-logfile-max-backups
agent-logfile-max-size

The following config settings configure the audit log files (note the missing “file” in the key name compared to the agent log file settings):

audit-log-max-backups
audit-log-max-size

The following config settings configure the model log files:

model-logfile-max-backups
model-logfile-max-size

Contributors: @hmlanigan, @jameinel, @manadart, @tmihoc

Last updated 10 days ago. Help improve this document in the forum or File an issue .

How to manage agent logs

Manage the logs

Stream the logs

Configure the logging level

Forward logs to an external logsink

Manage the log files

View the log files

Control the log file rotation