Integration

See also: Juju OLM | Integration

In Juju, an integration is a connection between applications, or between different units of the same application (the latter are also known as ‘peer relations’).

The definitions of relations are handled through interfaces, which are “loosely typed”, meaning there is no de-facto specification for:

• What information a relation must send/receive
• What actions are to be taken with data sent/received over the wire

While there is no formal specification, it is important to carefully consider the interface you present. Integrations are formed based on interface name only. For integrations to work effectively, the same data format needs to be set/observed on either side of the integration.

Contents:

• Defining integrations
• Peer integrations
• Integration and interface naming
• Implicit integrations
• Integration events
• Integration data
• Testing integrations

Defining integrations

Integrations between different applications are defined in a charm’s metadata.yaml using the provides and requires keywords. Integrations between multiple units of the same application are defined using the peers keyword.

Entries under the provides key define interfaces that the charm provides to other applications, those under the requires keyword define interfaces that the charm consumes from others. Despite their name, requires integrations are not always essential for the application to function – charm authors can indicate optionality in the metadata (see table below)…

Each integration defined under provides or requires is defined as a YAML map, where the top-level key is the name of the integration. The map’s key/value pairs define the options for the integration. There are four options defined in the metadata spec:

Example integration definitions can be seen below. For example, a database might specify:

name: charm-db
# ...
provides:
  database:
    interface: charm-db

And a simple web application might specify:

name: my-web-app
# ...
requires:
  database:
    interface: charm-db
    limit: 1
provides:
  website:
    interface: http
    optional: true

Together, these two metadata.yaml snippets indicate that an integration can be formed between the two respective charms. The database charm provides an integration named database with the charm-db interface, and similarly the web app charm requires an integration named database with the charm-db interface.

Note that the distinction between interface name and integration name enables developers to specify a name for the integration that makes the most sense in the context of their charm (which may be different from the name provided by the charm on the other end of the relation), while ensuring that the two are compatible where the interface name is the same. Using the above example, the database charm author could replace the name of the integration with charm-db, but the application charm would still be compatible, as the interface names remain the same.

Peer integrations

Charms can declare peer integrations, which causes each unit of an application to respond to the other units deployed in the same application. A peer integration is defined in exactly the same way as any other integration. For example, our fictional database could define a replicas integration, with the interface charm-db-replica

peers:
  replicas:
    interface: charm-db-replica

Peering integrations are particularly useful when your application supports clustering. Consider the implications of operating applications such as MongoDB, PostgreSQL, and ElasticSearch where clusters must exchange information amongst one another to perform proper clustering. Peer integration can be useful for coordinating operations against a clustered application without downtime – for example upgrading package versions or applying new configuration. Additionally, forming a peer integration is the only way to obtain a reliable IP address from other pods deployed for the same application, though this may be less relevant depending on the underlying substrate. This information is populated automatically for each unit using an implicit integration.

Integration and interface naming

The integration namespace is unrestricted with the exception that you may not provide an integration named juju, nor have its name begin with juju-. Charms attempting to provide integrations in this namespace will trigger an error.

Interface names are strings that must only contain characters a-z and -. Names must not start with -. The interface name is the only means of establishing whether two charms are compatible for relation; and carries with it nothing more than a mutual promise that the provider and requirer somehow know the communication protocol implied by the name.

Implicit integrations

Implicit integrations allow for simple integrations to be formed without requiring any modifications to target charms. Implicit integrations exist in the reserved namespace. There is currently only one such integration provided to all deployed applications: juju-info.

The juju-info integration captures very select data from the remote unit:

• private-address
• public-address

A common use for this type of relation is when authoring a subordinate charm that can be integrated with any other charm – in this case the juju-info implicit integration can be used. For example, the rsyslog-forwarder charm is a subordinate charm that requires a valid scope: container integration named logging. In the event that the principal charm doesn’t provide this, the logging charm author can use juju-info:

requires:
  logging:
    interface: logging-directory
    scope: container
  juju-info:
    interface: juju-info
    scope: container

The administrator can then issue the following command:

juju integrate some-app rsyslog-forwarder

If the some-app charm author doesn’t define the logging-directory interface which would configure the principal charm to log files into a directory, Juju will use the less-specific juju-info interface to create a config that configures the principal charm to forward syslog to the IP of the integrated application (using the information available from the juju-info implicit integration).

Integration events

Defining integrations is only the first step; for integrations to be useful, charm developers must implement handlers for the various events triggered by integrations, and ensure they implement the advertised interface.

Event types

The CharmBase class automatically defines 5 events per specified integration (where <name> is the name of the relation).

Integration data (‘databags’)

The primary means for applications to communicate over an integration is using integration data. This is exposed to the charm as a property of the event passed to the relevant callbacks. All integration events inherit from the RelationEvent class, which encapsulates the relevant Relation at <event>.relation. The Relation includes a data parameter of type RelationData which is a mapping containing all relation data stored by participating applications and units.

Integration data comprises a number of “databags” (or “data buckets”). Each application and each unit has its own databag, which is a Python str:str mapping. There are some rules about accessing integration data:

• Units can read and write their own unit databags (e.g. event.relation.data[self.unit])
• Units can read from any other unit databags (e.g. event.relation.data[event.unit])
• Application leaders can read and write their application databags (e.g. event.relation.data[self.app])
• Non-leaders can read from other application databags (e.g. event.relation.data[event.app])

A visualization:

Integration data is persisted to the controller database, and therefore persists across invocations of the charm code and through the various event handlers that may access it.

The framework also provides a mechanism for a charm author to fetch an integration from the model by name or ID. This can be useful in cases where methods (that are not integration event callbacks) need to access integration data. An example of this might be updating application data on a peer relation in the leader-elected event callback. Details of this method are documented in the API docs, and it is used in context in the examples provided. Where there are multiple integrations present for a single relation name, the get_relation method takes an id of the integration to fetch. In these cases it may be easier to iterate over the relations property of the model, which returns a Mapping of all integrations.

Examples

Requires/provides integration

Here we will construct a trivial example of an integration to bring these concepts to life.

For a given metadata.yaml:

requires:
  demo:
    interface: demo

We could implement the following callback for the demo_relation_changed event:

# ...
class SpecialCharm(CharmBase):
    # Set up some stored state in a private class variable
    _stored = StoredState()

    def __init__(self, *args):
        super().__init__(*args)
        # ...
        # Relation handling
        self.framework.observe(self.on.demo_relation_changed, self._on_demo_relation_changed)
        self.framework.observe(self.on.demo_relation_broken, self._on_demo_relation_broken)
        # Initialise our stored state
        self._stored.set_default(apps=dict())

    def _on_demo_relation_changed(self, event: RelationChangedEvent) -> None:
        # Do nothing if we're not the leader
        if not self.unit.is_leader():
            return

        # Check if the remote unit has set the 'leader-uuid' field in the
        # application data bucket
        leader_uuid = event.relation.data[event.app].get("leader-uuid")
        # Store some data from the relation in local state
        self._stored.apps.update({event.relation.id: {"leader_uuid": leader_uuid}})

        # Fetch data from the unit data bag if available
        if event.unit:
            unit_field = event.relation.data[event.unit].get("special-field")
            logger.info("Got data in the unit bag from the relation: %s", unit_field)

        # Set some application data for the remote application
        # to consume. We can do this because we're the leader
        event.relation.data[self.app].update({"token": f"{uuid4()}"})

        # Do something
        self._on_config_changed(event)

    def _on_demo_relation_broken(self, event: RelationBrokenEvent) -> None:
        # Remove the unit data from local state
        self._stored.apps.pop(event.relation.id, None)
        # Do something
        self._on_config_changed(event)
# ...

From the example above, note:

• The callback will only run if the current unit is the application leader.
• The callback gets some application data from the remote unit, and stores it locally in state.
• It fetches unit data from the remote unit that joined the relation.
• It sets some application data for itself, to be consumed by the remote application.

A corresponding application that could relate to this could specify the following metadata.yaml:

provides:
  demo:
    interface: demo

And the following integration event callback which:

• Sets some application data for the local application, if the unit is the leader
• Sets some unit data, containing just the unit’s name
• Fetches some application data from the remote application and stores it in state

def _on_demo_relation_changed(self, event: RelationChangedEvent) -> None:
        # If we're the current leader
        if self.unit.is_leader():
            # Set a field in the application data bucket
            event.relation.data[self.app].update({"leader-uuid": self._stored.uuid})

        # Set a field in the unit data bucket
        event.relation.data[self.unit].update({"special-field": self.unit.name})
        # Log if we've received data over the relation
        if self._stored.token == "":
            logger.info("Got a new token from '%s'", event.app.name)
            # Get some info from the relation and store it in state
            self._stored.token = event.relation.data[event.app].get("token")

Peer integration

This example illustrates a trivial peer integration, where the events can be observed using juju debug-log. The logging from the charm will demonstrate how each charm is notified of peers leaving/joining, and how the relation data is eventually consistent between units.

The basic premise behind this example is a simple clustered application that needs to present a single “master IP address” to another relation. Using the peer integration and the concept of application leadership, the IP address of the current leader is stored in the application data bucket on the peer relation, and cached locally into state on each of the followers. If a new leader is elected, the peer integration is fetched from the model and the application data stored on the peer relation is updated – triggering each of the other peers to update their local state.

The metadata.yaml for the example below contains the following:

peers:
  replicas:
    interface: charm-replica

And in the charm code:

# ...
class DemoCharm(CharmBase):
    """Charm the service."""

    _stored = StoredState()

    def __init__(self, *args):
        super().__init__(*args)
        # ...
        self.framework.observe(self.on.leader_elected, self._on_leader_elected)
        self.framework.observe(self.on.replicas_relation_joined, self._on_replicas_relation_joined)
        self.framework.observe(self.on.replicas_relation_departed, self._on_replicas_relation_departed)
        self.framework.observe(self.on.replicas_relation_changed, self._on_replicas_relation_changed)

        self._stored.set_default(leader_ip="")
        # ...

    def _on_leader_elected(self, event: LeaderElectedEvent) -> None:
        """Handle the leader-elected event"""
        logging.debug("Leader %s setting some data!", self.unit.name)
        # Get the peer relation object
        peer_relation = self.model.get_relation("replicas")
        # Get the bind address from the juju model
        # Convert to string as relation data must always be a string
        ip = str(self.model.get_binding(peer_relation).network.bind_address)
        # Update some data to trigger a replicas_relation_changed event
        peer_relation.data[self.app].update({"leader-ip": ip})

    def _on_replicas_relation_joined(self, event: RelationJoinedEvent) -> None:
        """Handle relation-joined event for the replicas relation"""
        logger.debug("Hello from %s to %s", self.unit.name, event.unit.name)

        # Check if we're the leader
        if self.unit.is_leader():
            # Get the bind address from the juju model
            ip = str(self.model.get_binding(event.relation).network.bind_address)
            logging.debug("Leader %s setting some data!", self.unit.name)
            event.relation.data[self.app].update({"leader-ip": ip})

        # Update our unit data bucket in the relation
        event.relation.data[self.unit].update({"unit-data": self.unit.name})

    def _on_replicas_relation_departed(self, event: RelationDepartedEvent) -> None:
        """Handle relation-departed event for the replicas relation"""
        logger.debug("Goodbye from %s to %s", self.unit.name, event.unit.name)

    def _on_replicas_relation_changed(self, event: RelationChangedEvent) -> None:
        """Handle relation-changed event for the replicas relation"""
        logging.debug("Unit %s can see the following data: %s", self.unit.name, event.relation.data.keys())
        # Fetch an item from the application data bucket
        leader_ip_value = event.relation.data[self.app].get("leader-ip")
        # Store the latest copy locally in our state store
        if leader_ip_value and leader_ip_value != self._stored.leader_ip:
            self._stored.leader_ip = leader_ip_value


if __name__ == "__main__":
    main(DemoCharm)
# ...

To illustrate how these events play out, open a separate terminal and run juju debug-log. Use the juju CLI to add units to the charm, observing the events in the debug log, then remove them and see the reverse.

Handling integration data from non-relation hooks

Sometimes you have to manipulate integration data in a context where there is no “current” integration; in other words, when a non-relation-related event occurs. For example, when a config-changed event occurs. Suppose the charm has the same demo interface as above.

def _on_config_changed(self, event: ConfigChangedEvent):
    # from the Model, you can access all relations your charm has at the moment, 
    # by endpoint name.
    my_relations: Mapping[str, List[Relation]] = self.model.relations
    
    # the list of relations bound to this charm's `demo` endpoint.
    demo_relations: List[Relation] = my_relations['demo']
    
    for relation in demo_relations:
        # this is how you get all of the databags in this relation.
        # RelationData is a mapping from the "owner" of a databag to the databag's contents..
        data: RelationData = relation.data
        
        # LOCAL ENTITIES:

        # so to get the app databag of **this** application, you can:
        this_app_data: RelationDataContents = data[self.app]
        
        # and to get the unit databag of **this** unit:
        this_unit_data: RelationDataContents = data[self.unit]
        
        # we can get a list of our peer units like so:
        peers: List[Unit] = [u for u in relation.units if u.app is self.app]

        # and now we can get to the unit databags of the other units of **this** app:
        peer_databags: List[RelationDataContents] = [relation.data[peer] for peer in peers]
        
        # REMOTE ENTITIES:
        
        # to get the app databag of this relation's remote side:
        remote_app_data: RelationDataContents = relation.data[relation.app]
        
        # to get a list of the remote units:
        remote_units: List[Unit] = [u for u in relation.units if u.app is not self.app]
        
        # and now we can get to the unit databags of the other **remote** units:
        remote_unit_databags: List[RelationDataContents] = [relation.data[rem] for rem in remote_units]

Testing integrations

Integrations are fundamental to Juju’s model-driven approach, and therefore they should be tested rigorously to ensure they behave as expected, and that charms present the right interfaces to other charms.

Harness provides a number of methods to aid testing relations, which are demonstrated below:

# ...
harness = Harness(MyCharm)
# Do initial setup here - note that adding a relation does not trigger any events
# in the context of Harness. While this differs from Juju, it makes testing simpler
# It takes the relation name, and remote application as arguments
relation_id = harness.add_relation('db', 'postgresql')
# This will trigger a relation-joined event, it takes the relation id,
# and name of the unit to join to the relation
harness.add_relation_unit(relation_id, 'postgresql/0')
# This method both updates the relation data, and triggers a relation-changed event
# This takes the relation id, name of the application or unit, and a value
harness.update_relation_data(relation_id, 'postgresql/0', {'key': 'val'})
harness.set_leader(True)
harness.update_config({'initial': 'config'})
# Trigger the same events Juju would when deploying/installing charm.

# Specifically here: install, db-relation-created('postgresql'), leader-elected, config-changed, start, db-relation-joined('postgresql/0'), db-relation-changed('postgresql/0')
harness.begin_with_initial_hooks()
# ...

The final method regarding integration testing is get_relation_data() which returns the relation data bucket for a given application or unit. Example usage:

# ...
self.assertEqual(self.harness.get_relation_data(relation_id, "postgresql/0"), {
    "hostname": "dbhost",
    "port": "4444",
    "user": "charmuser",
    "pass": "supersecret"
})
# ...

Note that when calling get_relation_data or update_relation_data you are able to read/write relation data for both the charm being tested, but also the ‘remote’ charm in the relation. This is essential to ensure that your charm responds to remote changes in relation data correctly, though can become difficult where interfaces are complicated or subject to change (perhaps during development).

In both cases, the get_relation_data and update_relation_data take an argument to specify which charm is having it’s data read or changed – even though one or more of those charms may be entirely simulated due to the nature of Harness.

Last updated 4 months ago.