How to write an integration test for a charm

This document covers writing integration tests, smoke tests, and any other tests that might benefit from being run against a live environment.

Integration tests should not be the first thing that charm authors write. The Operator Framework has an excellent testing harness that makes it possible to write unit tests with minimal mocking, and a reasonable expectation that code that passes harness tests can be successfully executed in an actual Juju model.

But integration tests can still be tremendously useful! This document covers setting up, writing, and running a simple Smoke test. The tools and information outlined here can then be used to create more expansive suites of integration tests.

Setup

By convention, integration tests are kept in the charm’s source tree, in a directory called tests/integration. The tests here utilize pytest-operator, which is a lightweight library that wraps python-libjuju with some testing related convenience functions. With pytest-operator, it is easy to setup and tear down temporary models to hold your tests.

The following lines in a charm’s tox.ini will setup integration tests.

[testenv:integration]
description = Run integration tests
deps =
     juju
     pytest
     pytest-operator
     ipdb
commands =
    pytest -v --tb native --ignore={[vars]tst_dir}unit --log-cli-level=INFO -s {posargs}

This document assumes that charm authors have installed python tox, charmcraft, and juju, and have bootstrapped a controller.

An example smoke test.

Here is the simplest possible implementation of an integration test – a smoke test that builds and deploys the charm, and verifies that the installation hooks finish executing without error:

import logging

from pytest_operator.plugin import OpsTest

logger = logging.getLogger(__name__)


@pytest.mark.abort_on_fail
async def test_smoke(ops_test: OpsTest):
    charm = await ops_test.build_charm(".")
    app = await ops_test.model.deploy(charm)
    unit0 = app.units[0]

    await ops_test.model.block_until(lambda: app.status in ("active", "error"), timeout=60)
    assert app.status, "active"

Let’s break that down.

@pytest.mark.abort_on_fail

When pytest runs a test module, it will execute all the routines in that module with test in their name, in the order that they appear in the module. We can take advantage of this to add a single “setup” test at the top of the file. This test will execute before the other tests in the module. The abort_on_fail decorator will abort test execution if something goes wrong.

Note:

If two tests should not share a model, simply put them in a different file, with a separate setup.
For a more object oriented approach, charm authors may use IsolatedAsyncioTestCase from Python’s unittest library. See the smoke tests in charm-rolling-ops for an example.
This example module contains only one test, which serves as both setup and smoke test.

async def test_smoke(ops_test: OpsTest)

pythonlibjuju uses Python’s asyncio library to interact with the Juju model. Our tests are linear, but we do need to define our test routines as asynchronous routines, and use await when calling functions that execute asynchronously.

charm = await ops_test.build_charm(".")

This line will build the charm with charmcraft. This document assumes that charmcraft is setup on the developer’s machine, and that the charm can be built.

app = await ops_test.model.deploy(charm)

ops_test.model is simply an instance of python-libjuju's Model class. Charm authors may reference the python-libjuju docs for a complete list of capabilities and usage.

The call to Model.deploy will result in the freshly built charm being deployed to the test model.

Model.deploy returns an Application object. In other tests, this Application object may be re-referenced with app = ops_test.model.applications[<charm-name>].

await ops_test.model.block_until(lambda: app.status in ("active", "error", "blocked"), timeout=60)

We now wait until the app status is either active or in an error state. All apps start out in with a maintenance, status, so it is sufficient, for our smoke test, to simply wait for that initial status to change. We timeout so that our tests don’t hang on stalls.

Note: there is a helper in pytest-ops to address this particular scenario more completely. We don’t use it here, because this doc is intended to give charm authors ideas for expanding on the basic smoke test. But we could replace the last two lines of our code with the following:

ops_test.model.wait_for_idle(apps=['application_name'], status='active', timeout=60, wait_for_exact_units=1)

assert app.status == "active"

We complete this simple smoke test by verifying that the status is active, meaning that the charm appears to have deployed and installed correctly.

How to write an integration test for a charm

Setup

An example smoke test.

Further Reading