There are various logging resources available to the Juju operator.
This page covers the following topics:
Model logs can be considered as Juju’s “regular logs” and are intended to be inspected with the
debug-log command. This method provides logs on a per-model basis and is therefore more convenient than reading individual logs on multiple (Juju) machines directly on the file system. The latter can nonetheless be done in exceptional circumstances and some explanation is provided here.
See Controller HA and logging when viewing logs in an HA context.
One of the roles of the Juju agent, ‘jujud’, is to perform logging.
The debug-log command
debug-log command shows the consolidated logs of all Juju agents (machines and units) running in a model. The
switch command is used to change models. Alternatively, a model can be chosen with the ‘-m’ option. The default model is the current model.
The ‘controller’ model captures logs related to internal management (controller activity) such as adding machines and services to the database. Whereas a hosted model will provide logs concerning activities that take place post- provisioning.
Due to the above, when deploying a service, it is normal for there to be an absence of logging in the workload model since logging first takes place in the ‘controller’ model.
The output is a fixed number of existing log lines (number specified by possible options; the default is 10) and a stream of newly appended messages. Both existing lines and appended lines can be filtered by specifying options.
The exception to the streaming is when limiting the output (option ‘–limit’; see below) and that limit is attained. In all other cases the command will need to be interrupted with ‘Ctrl-C’ in order to regain the shell prompt.
For complete syntax, see the Command reference page.
To begin with the last ten log messages:
To begin with the last thirty log messages:
juju debug-log -n 30
To begin with all the log messages:
juju debug-log --replay
To begin with the last twenty log messages for the ‘lxd-pilot’ model:
juju debug-log -m lxd-pilot -n 20
To begin with the last 500 lines. The
grep utility is used as a text filter:
juju debug-log -n 500 | grep amd64
Juju logging is hierarchical and can be granular.
<root> refers to all logs related to a juju machine, including controller operation when applied to the controller model.
unit refers to all logs related to a juju unit.
#label-name allows for logging based on a topic. Currently available topics are:
- cmr: cross model relations
- cmr-auth: cross model relations authorization
- charmhub: dealing with the charmhub client and callers
- http: HTTP requests
- metrics: juju metrics, this should be used as a fallback for when prometheus isn’t available.
These are modules of the juju code base. See advanced filtering for more information.
You can verify the current logging level like this:
juju model-config logging-config
Output will be similar to the following:
The above is the default configuration. It sets the machine agent log level at ‘WARNING’, the unit agent log level at ‘DEBUG’, and the http label to TRACE.
The logging levels, from most verbose to least verbose, are as follows:
Change the logging level
When diagnosing an issue (and possibly filing a bug), the first step is to make logging more verbose. For instance, to increase the logging level of the unit agents to ‘TRACE’ you would enter the following command:
juju model-config logging-config="<root>=WARNING;unit=TRACE"
To avoid filling up the database unnecessarily, when verbose logging is no longer needed, do not forget to return logging to normal levels.
The logging level can also be changed on a per-unit basis. See section Agent logging override below.
A Juju log line is written in this format:
<entity> <timestamp> <log-level> <module>:<line-no> <message>
--exclude options select and deselect, respectively, the entity that logged the message. An entity is a Juju machine or unit agent. The entity names are similar to the names shown by
--exclude-module options can be used to influence the type of message displayed based on a (dotted) module name. The module name can be truncated.
--exclude-label options can be used to influence the type of message displayed based on the labels defined in logging granularity. No hash is required.
A combination of machine and unit filtering uses a logical OR whereas a combination of module and machine/unit filtering uses a logical AND.
--level option places a limit on logging verbosity (e.g.
--level INFO will allow messages of levels ‘INFO’, ‘WARNING’, and ‘ERROR’ to be shown).
To begin with the last 1000 lines and exclude messages from machine 3:
juju debug-log -n 1000 --exclude machine-3
To select all the messages emitted from a particular unit and a particular machine in the entire log:
juju debug-log --replay --include unit-mysql-0 --include machine-1
The unit can also be written ‘mysql/0’ (as shown by
To see all WARNING and ERROR messages in the entire log:
juju debug-log --replay --level WARNING
To see all logs on the cmr topic (label):
juju debug-log --include-label cmr
To progressively exclude more content from the entire log:
juju debug-log --replay --exclude-module juju.state.apiserver juju debug-log --replay --exclude-module juju.state juju debug-log --replay --exclude-module juju
To begin with the last 2000 lines and include messages pertaining to both the juju.cmd and juju.worker modules:
juju debug-log --lines 2000 \ --include-module juju.cmd \ --include-module juju.worker
Agent logging override
As we’ve seen, the logging level for machine agents and unit agents are specified as a single model configuration setting. However, in some situations (e.g. targeted verbose debugging) it may be desirable to increase the logging level on a per-agent basis. This can also be done after having reduced the model-wide log level (as explained in section Change the logging level above).
For example, let’s enable ‘TRACE’ logging level to a unit of MySQL. We begin by logging in to the unit’s machine (‘0’ in this example):
juju ssh mysql/0
/var/lib/juju/agents/unit-mysql-0/agent.conf is then edited by adding the line
LOGGING_OVERRIDE: juju=trace to the ‘values’ section.
To be clear, the bottom of the file now looks like:
loggingconfig: <root>=WARNING;unit=DEBUG values: CONTAINER_TYPE: "" NAMESPACE: "" LOGGING_OVERRIDE: juju=trace mongoversion: "0.0"
The affected agent will need to be restarted to have this change take effect:
sudo systemctl restart jujud-unit-mysql-0.service
Log files are located on every machine Juju creates, including the controller. They reside under
/var/log/juju and correspond to the machine and any units.
Using the example from a previous section:
juju ssh 2 ls -lh /var/log/juju
-rw------- 1 syslog syslog 22K Apr 28 00:42 machine-2.log -rw------- 1 syslog syslog 345K Apr 28 16:58 unit-nfs2-0.log
There is one extra log file on each controller:
juju ssh -m controller 0 ls -lh /var/log/juju
-rw------- 1 syslog syslog 2.3M Apr 28 17:05 logsink.log -rw------- 1 syslog syslog 85K Apr 28 17:03 machine-0.log
logsink.log contains logs for all models managed by the controller. Its contents get sent to the database where it is consumed by the
In a High availability scenario,
logsink.log is not guaranteed to contain all messages since agents have a choice of several controllers to send their logs to. The
debug-log command should be used for accessing consolidated data across all controllers.
On a per-model basis log messages can optionally be forwarded to a remote syslog server over a secure TLS connection.
See Rsyslog documentation for help with security-related files (certificates, keys) and the configuration of the remote syslog server.
Remote logging is configured during the controller-creation step by supplying a YAML format configuration file:
juju bootstrap <cloud> --config logconfig.yaml
The contents of the YAML file is of the form:
syslog-host: <host>:<port> syslog-ca-cert: | -----BEGIN CERTIFICATE----- <cert-contents> -----END CERTIFICATE----- syslog-client-cert: | -----BEGIN CERTIFICATE----- <cert-contents> -----END CERTIFICATE----- syslog-client-key: | -----BEGIN PRIVATE KEY----- <cert-contents> -----END PRIVATE KEY-----
To actually enable remote logging for a model a configuration key needs to be set for that model:
juju model-config -m <model> logforward-enabled=True
An initial 100 (maximum) existing log lines will be forwarded.
See Configuring models for extra help on configuring a model.
Note that it is possible to configure and enable forwarding on all the controller’s models in one step:
juju bootstrap <cloud> --config logforward-enabled=True --config logconfig.yaml
Juju audit logging provides a chronological account of all events by capturing invoked user commands. These logs reside on the controller involved in the transmission of commands affecting the Juju client, Juju machines, and the controller itself.
The audit log filename is
/var/log/juju/audit.log and contains records which are either:
- a Conversation, a collection of API methods associated with a single top-level CLI command
- a Request , a single API method
- a ResponseErrors, errors resulting from an API method
Information can be filtered out of the audit log to prevent its file(s) from growing without bounds and making it difficult to read. See Excluding information from the audit log.
The log is typically viewed by connecting to the controller over SSH and looking at the file:
juju ssh -m controller 0 more /var/log/juju/audit.log
Log file sizes and rotation
Juju has settings to control the maximum size and rotation of the various log files it produces.
There are two settings for each log file type: maximum size, and number of backups. When the current log file of a particular type reaches the maximum size, Juju renames the log file to include a timestamp and gzips it, producing a “backup” log file.
Here’s an example of the controller’s machine agent logs with the maximum size set to 1MB, showing two timestamped backups as well as the current log file:
$ juju bootstrap localhost --config agent-logfile-max-size=1MB $ lxc exec juju-6bf629-0 -- ls -l /var/log/juju ... -rw-r----- 1 syslog adm 3577 Jan 12 02:01 machine-0-2022-01-12T02-01-07.995.log.gz -rw-r----- 1 syslog adm 3578 Jan 12 02:02 machine-0-2022-01-12T02-02-08.011.log.gz -rw-r----- 1 syslog adm 600000 Jan 12 02:02 machine-0.log
The full list of the controller settings that configure log file rotation is shown below. Normally these are set at bootstrap time with the
--config option (see How to configure a controller).
Configuring agent log files
The following config settings configure agent log files, including the API server “log sink”, the machine agent logs on controller and unit machines, and the unit agent logs:
Configuring audit log files
The following config settings configure the audit log files (note the missing “file” in the key name compared to the agent log file settings):
Configuring model log files
The following config settings configure the model log files:
Last updated 2 months ago.