How to manage machines

See also: Machine

This document shows how to manage machines.

Contents:

Add a machine

To add a new machine to a model, run the add-machine command, as below. juju will start a new machine by requesting one from the cloud provider.

juju add-machine

The command also provides many options. By using them you can customize many things. For example, you can provision multiple machines, specify the Ubuntu series to be installed on them, choose to deploy on a lxd container inside a machine, apply various constraints to the machine (e.g., storage, spaces, …) to override more general defaults (e.g., at the model level), etc.

Machines provisioned via add-machine can be used for an initial deployment (deploy ) or a scale-out deployment (add-unit).

See more: juju add-machine

To add a machine to a model, in your Terraform plan add a resource of the juju_machine type, specifying the model.

resource "juju_machine" "machine_0" {
  model       = juju_model.development.name
}

You can optionally specify a base, a name, regular constraints, storage constraints, etc. You can also specify a private_key_file, public_key_file, and ssh_address – that will allow you to add to the model an existing, manual machine (rather than a virtual one provisioned for you by the cloud).

See more: juju_machine (resource)

To add a machine to a model, on a connected Model object, use the add_machine() method. For example:

await my_model.add_machine()

See more: add_machine(), Model (module)

Troubleshooting:

Issue during machine provisioning can occur at any stage in the following sequence:

Provision resources/a machine M from the relevant cloud, via cloud-init maybe network config, download the jujud binaries from the controller, start jujud.

To troubleshoot, try to gather more information until you understand what caused the issue.

See more: How to troubleshoot your deployment

Retry provisioning for a failed machine

To retry provisioning (a ) machine(s) (e.g., for a failed deploy, add-unit, or add-machine), run the retry-provisioning command followed by the (space-separated) machine ID(s). For example:

juju retry-provisioning 3 27 57

See more: juju retry-provisioning

The terraform juju client does not support this. Please use the juju client.

The python-libjuju client does not currently support this. Please use the juju client.

List all machines

To see a list of all the available machines, use the machines command:

juju machines

The output should be similar to the one below:

Machine  State    Address         Inst id        Base          AZ  Message
0        started  10.136.136.175  juju-552e37-0  ubuntu@22.04      Running
1        started  10.136.136.62   juju-552e37-1  ubuntu@22.04      Running

See more: juju machines

The terraform juju client does not support this. Please use the juju client.

To see a list the names of all the available machines on a model, on a connected Model object, use the get_machines() method. For example:

await my_model.get_machines()

To get a list of the machines as Machine objects on a model, use the machines property on the Model object. This allows direct interaction with any of the machines on the model. For example:

machines = my_model.machines
my_machine = machines[0] # Machine object
print(my_machine.status)

See more: get_machines(), Model (module), Model.machines (property), Machine (object)

View details about a machine

To see details about a machine, use the show-machine command followed by the machine ID. For example:

juju show-machine 0
Expand to see a sample output
model: localhost-model
machines:
  "0":
    juju-status:
      current: started
      since: 27 Oct 2022 09:37:17+02:00
      version: 3.0.0
    hostname: juju-552e37-0
    dns-name: 10.136.136.175
    ip-addresses:
    - 10.136.136.175
    instance-id: juju-552e37-0
    machine-status:
      current: running
      message: Running
      since: 27 Oct 2022 09:36:10+02:00
    modification-status:
      current: applied
      since: 27 Oct 2022 09:36:07+02:00
    base:
      name: ubuntu
      channel: "22.04"
    network-interfaces:
      eth0:
        ip-addresses:
        - 10.136.136.175
        mac-address: 00:16:3e:cc:f2:16
        gateway: 10.136.136.1
        space: alpha
        is-up: true
    constraints: arch=amd64
    hardware: arch=amd64 cores=0 mem=0M

See more: juju show-machine

The terraform juju client does not support this. Please use the juju client.

To see details about a machine, on a connected Model object, get a hold of the Machine object within the model using the machines property. This allows direct interaction with the machine, such as accessing all the details (via the object properties) for that machine. For example:

my_machine = await my_model.machines[0]
# Then we can access all the properties to view details
print(my_machine.addresses)
print(my_machine.agent_version)
print(my_machine.hostname)
print(my_machine.status)

See more: Model (module), Model.machines (property), Machine (object)

Show the status of a machine

To see the status of a machine, use the status command:

juju status

This will report the status of the model, its applications, its units, and also its machines.

Expand to see a sample output
Model            Controller            Cloud/Region         Version  SLA          Timestamp
localhost-model  localhost-controller  localhost/localhost  3.0.0    unsupported  13:51:33+02:00

App       Version  Status   Scale  Charm     Channel  Rev  Exposed  Message
influxdb           waiting    0/1  influxdb  stable    24  no       waiting for machine

Unit        Workload  Agent       Machine  Public address  Ports  Message
influxdb/0  waiting   allocating  2                               waiting for machine

Machine  State    Address  Inst id  Base          AZ  Message
2        pending           pending  ubuntu@20.04      Retrieving image: rootfs: 4% (10.87MB/s)

See more: juju status

The terraform juju client does not support this. Please use the juju client.

To see the status of a machine, on a connected Model object, get a hold of the Machine object within the model using the machines property. The status is then retrieved directly via the Machine object properties, in this case the status property. For example:

my_machine = await my_model.machines[0]
print(my_machine.status)

See more: Model (module), Model.machines (property), Machine (object), Machine.status (property)

Manage constraints for a machine

See also: Constraint

Set values. You can set constraint values for an individual machine when you create it manually, by using the add-machine command with the constraints flag followed by a quotes-enclosed list of your desired key-value pairs, for example:

juju add-machine --constraints="cores=4 mem=16G"
Expand to see the command along with its result, including constraints
$ juju add-machine --constraints="cores=4 mem=16G"
created machine 0
$ juju show-machine 0
model: test
machines:
  "0":
    juju-status:
      current: pending
      since: 20 Mar 2023 12:58:52+01:00
    instance-id: pending
    machine-status:
      current: pending
      since: 20 Mar 2023 12:58:52+01:00
    modification-status:
      current: idle
      since: 20 Mar 2023 12:58:52+01:00
    base:
      name: ubuntu
      channel: "22.04"
    constraints: cores=4 mem=16384M

See more: juju add-machine --constraints, How to add a machine

Get values. You can get constraint values for for an individual machine by viewing details about the command with the show-machine command, for example:

juju show-machine 0
Expand to see a sample output, including constraints
model: controller
machines:
  "0":
    juju-status:
      current: started
      since: 01 Mar 2023 15:08:34+01:00
      version: 3.1.0
    hostname: juju-6a1e1b-0
    dns-name: 10.136.136.239
    ip-addresses:
    - 10.136.136.239
    instance-id: juju-6a1e1b-0
    machine-status:
      current: running
      message: Running
      since: 07 Feb 2023 13:53:20+01:00
    modification-status:
      current: applied
      since: 20 Mar 2023 08:53:12+01:00
    base:
      name: ubuntu
      channel: "22.04"
    network-interfaces:
      enp5s0:
        ip-addresses:
        - 10.136.136.239
        mac-address: 00:16:3e:4d:aa:69
        gateway: 10.136.136.1
        space: alpha
        is-up: true
    constraints: virt-type=virtual-machine
    hardware: arch=amd64 cores=0 mem=0M virt-type=virtual-machine
    controller-member-status: has-vote

See more: juju show-machine, How to view details about a machine

To set constraints for a machine, in your Terraform plan, in the machine resource definition, set the constraints attribute to the desired quotes-enclosed, space separated list of key=value pairs. For example:

resource "juju_machine" "machine_0" {
  model       = juju_model.development.name
  name        = "machine_0"
  constraints = "tags=my-machine-tag"
}

See more: juju_machine (resource)

Set values. To set constraint values for an individual machine when you create it manually, on a connected Model, use the add_machine() method, passing constraints as a parameter. For example:

machine = await model.add_machine(
            constraints={
                'arch': 'amd64',
                'mem': 256 * MB,
            })

Get values. The python-libjuju client does not currently support getting constraint values for for an individual machine. However, to retrieve machine constraints on a model, on a connected Model, use the get_constraints() method. For example:

await my_model.get_constraints()

Note that this will return None if no constraints have been set on the model.

See more: add_machine(), get_constraints(), Model (module)

Execute a command inside a machine

To run a command in a machine, use the exec command followed by the target machine(s) (--all, --machine, --application or --unit) and the commands you want to run. For example:


# Run the 'echo' command in the machine corresponding to unit 0 of the 'ubuntu' application:
juju exec --unit ubuntu/0 echo "hi"

# Run the 'echo' command in all the machines corresponding to the 'ubuntu' application:
 juju exec --application ubuntu echo "hi"

The exec command can take many other flags, allowing you to specify an output file, run the commands sequentially (since juju v.3.0, the default is to run them in parallel), etc.

See more: juju exec (before juju v.3.0, juju run)

The terraform juju client does not support this. Please use the juju client.

To run a command in a machine, on a Machine object, use the ssh() method, passing a command as a parameter. For example:

output = await my_machine.ssh("echo test")
assert 'test' in output

To run a command in all the machines corresponding to an application, on an Application object, use the run() method, passing the command as a parameter. For example:

output = await my_application.run("echo test")
assert 'test' in output

See more: ssh(), Machine (object), run(), Application (object)

Access a machine via SSH

There are two ways you can connect to a Juju machine: via juju ssh or via a standard SSH client. The former is more secure as it allows access solely from a Juju user with admin model access.



Use the juju ssh command

First, make sure you have admin access to the model and your public SSH key has been added to the model.

If you are the model creator, your public SSH key is already known to juju and you already have admin access for the model. If you are not the model creator, see How to manage users and User access levels for how to gain admin access to a model and How to manage SSH keys for how to add your SSH key to the model.

Then, to initiate an SSH session or execute a command on a Juju machine (or container), use the juju ssh command followed by the target machine (or container). This target can be specified using a machine (or container) ID or using the ID of the unit that it hosts. Both can be retrieved from the output of juju status. For example, below we ssh into machine 0 and inside of it run echo hello:

juju ssh 0 echo hello 

By passing further arguments and options, you can also run this on behalf of a different qualified user (other than the current user) or pass a private SSH key instead.

See more: juju ssh

Use the OpenSHH ssh command

First, make sure you’ve added a public SSH key for your user to the target model.

If you are the model creator, your public SSH key is already known to juju and you already have admin access for the model. If you are not the model creator, see How to manage users and User access levels for how to gain admin access to a model and How to manage SSH keys for how to add your SSH key to the model.

Alternatively, for direct access using a standard SSH client, it is also possible to add the key to an individual machine using standard methods (manually copying a key to the authorized_keys file or by way of a command such as ssh-import-id in the case of Ubuntu).

Then, to connect to a machine via the OpenSSH client, use the OpenSSH ssh command followed by <user account>@<machine IP address>, where the default user account added to a Juju machine, to which public SSH keys added by add-ssa-key or import-ssh-key, is ubuntu. For example, for a machine with an IP address of 10.149.29.143, do the following:

ssh ubuntu@10.149.29.143

See more: OpenSSH

The terraform juju client does not support this. Please use the juju client.

The python-libjuju client does not currently support this. Please use the juju client.

Copy files securely between machines

The scp command copies files securely to and from machines.

Options specific to scp must be preceded by double dashes: --.

Examples:

Copy 2 files from two MySQL units to the local backup/ directory, passing -v to scp as an extra argument:

juju scp -- -v mysql/0:/path/file1 mysql/1:/path/file2 backup/

Recursively copy the directory /var/log/mongodb/ on the first MongoDB server to the local directory remote-logs:

juju scp -- -r mongodb/0:/var/log/mongodb/ remote-logs/

Copy a local file to the second apache2 unit in the model “testing”. Note that the -m here is a Juju argument so the characters -- are not used:

juju scp -m testing foo.txt apache2/1:

Juju cannot transfer files between two remote units because it uses public key authentication exclusively and the native (OpenSSH) scp command disables agent forwarding by default. Either the destination or the source must be local (to the Juju client).

See more: juju scp

The terraform juju client does not support this. Please use the juju client.

To copy files securely between machines, on a Machine object, use the scp_to() and scp_from() methods, passing source and destination parameters for the transferred files or directories. For example:

# Transfer from local machine to Juju machine represented by my_machine object
with open(file_name, 'r') as f:
    await my_machine.scp_to(f.name, 'testfile')

# Transfer from my_machine to local machine
with open(file_name, 'w') as f:
    await my_machine.scp_from('testfile', f.name)
    assert f.read() == b'contents_of_file'

# Pass -r for recursively copy a directory via the `scp_opts` parameter.
await my_machine.scp_to('my_directory', 'testdirectory', scp_opts=['-r'])

See more: scp_to(), scp_from(), Machine (object)

Upgrade a machine

See also: Upgrading things

The process for how to upgrade a machine depends on whether the machine is a controller machine or a workload machine.

Upgrade a controller machine

You cannot upgrade a controller machine. Instead, you must bootstrap a new controller with the intended base, migrate your old models to it, configure the models to the new base, and then remove the old controller, as shown below.

1. Create a new controller with the intended base. To create a new controller with a base of your choice, run the bootstrap command with the bootstrap-base flag. For example:

juju bootstrap aws aws-new --bootstrap-base=<base>

See more: How to create a controller, juju bootstrap

2. Migrate your existing models to the new controller.

See more: How to migrate a workload model to another controller

3. Configure the migrated models such that all new machines have the new base.

juju model-config -m <model name> default-base=<base>

4. Destroy the old controller.

juju destroy-controller aws-old

Upgrade a workload machine

Support for upgrading the base for an individual machine after it has been provisioned will be removed starting with Juju 4. See more: Discourse | Juju 4.0 to remove upgrade-machine.

1. Tell Juju to take the machine out of circulation, in preparation for an upgrade. To achieve this, run the upgrade-machine command followed by the machine ID, the subcommand prepare, and the base to upgrade to. For example, the code below tells juju to prepare machine 3 for an upgrade to ubuntu@22.04.

Once the prepare command has been issued, there is no way to cancel or abort the process. Once you commit to prepare you must complete the process or you will end up with an unusable machine!

If you’re using Juju <3.1: Instead of a base you must specify the series. Thus, not ubuntu@22.04 but rather jammy.

juju upgrade-machine 3 prepare ubuntu@22.04

This has multiple effects:

  1. The machine is no longer available for charm deployments or for hosting new containers.
  2. Juju prepares the machine for the upcoming OS upgrade. All units on the machine are taken into account.

2. Perform the upgrade. This is done manually. On an Ubuntu-based machine, you can do this by logging in to the machine via SSH and executing the do-release-upgrade command:

juju ssh 3
$ do-release-upgrade

Make sure to reserve some time for maintenance, in case any issues arise.

3. Tell Juju that the machine has been successfully upgraded. To achieve this, run the upgrade-machine command with the complete subcommand.

juju upgrade-machine 3 complete

Done! The upgraded machine is again available for charm deployments.

See more: juju upgrade-machine (before Juju 3, upgrade-series)

The terraform juju client does not support this. Please use the juju client.

The python-libjuju client does not currently support this. Please use the juju client.

Remove a machine

See also: Removing things

To remove a machine, use the remove-machine command followed by the machine ID. For example:

juju remove-machine 3

It is not possible to remove a machine that is currently hosting either a unit or a container. Either remove all of its units (or containers) first or, as a last resort, use the --force option.

In some situations, even with the --force option, the machine on the backing cloud may be left alive. Examples of this include the Manual cloud or if harvest provisioning mode is not set (see provisioner-harvest-mode). In addition to those situations, if the client has lost connectivity with the backing cloud, any backing cloud, then the machine may not be destroyed, even if the machine’s record has been removed from the Juju database and the client is no longer aware of it.

By using various options, you can also customize various other things, for example, the model or whether to keep the running cloud instance or not.

See more: juju remove-machine

To remove a machine, remove its resource definition from your Terraform plan.

See more: juju_machine (resource)

To remove a machine, on a Machine object, use the destroy() method. For example:

await my_machine.destroy()

See more: destroy(), Machine (object)


Contributors: @alhama7a, @cderici, @tmihoc