Integration
See first: Juju | Integration
See also: How to add an integration to a charm
Before juju v.3.0
, ‘integrations’ were called ‘relations’. Remnants of this persist in the names, options, and output of certain commands, and in integration event names.
In Juju, an integration is a connection between applications, or between different units of the same application (the latter are also known as ‘peer relations’).
The definitions of relations are handled through interfaces, which are “loosely typed”, meaning there is no de-facto specification for:
- What information a relation must send/receive
- What actions are to be taken with data sent/received over the wire
While there is no formal specification, it is important to carefully consider the interface you present. Integrations are formed based on interface name only. For integrations to work effectively, the same data format needs to be set/observed on either side of the integration.
Contents:
- Defining integrations
- Peer integrations
- Integration and interface naming
- Implicit integrations
- Integration events
- Integration data
- Testing integrations
Defining integrations
Integrations between different applications are defined in a charm’s metadata.yaml
using the provides
and requires
keywords. Integrations between multiple units of the same application are defined using the peers
keyword.
Entries under the provides
key define interfaces that the charm provides to other applications, those under the requires
keyword define interfaces that the charm consumes from others. Despite their name, requires
integrations are not always essential for the application to function – charm authors can indicate optionality in the metadata (see table below)…
Each integration defined under provides
or requires
is defined as a YAML map, where the top-level key is the name of the integration. The map’s key/value pairs define the options for the integration. There are four options defined in the metadata spec:
Field | Type | Default | Required? | Description |
---|---|---|---|---|
interface |
string |
yes | Name of the integration interface | |
limit |
int |
nil |
no | Maximum number of connections permitted to endpoint |
optional |
bool |
false |
no | True if the integration is optional for the charm deployment. This field is ignored by Juju, and is used to indicate optionality to administrators. |
scope |
string |
global |
no | Scope of the integration. Options are global or container . This option controls the set of units from integrated applications that are reported to the unit as members of the integration. Container-scoped integrations are restricted to reporting details of a single principal unit to a single subordinate, and vice versa, while global integrations consider all possible remote units. Subordinate charms are only valid if they have at least one requires integration with container scope. |
Example integration definitions can be seen below. For example, a database might specify:
name: charm-db
# ...
provides:
database:
interface: charm-db
And a simple web application might specify:
name: my-web-app
# ...
requires:
database:
interface: charm-db
limit: 1
provides:
website:
interface: http
optional: true
Together, these two metadata.yaml
snippets indicate that an integration can be formed between the two respective charms. The database charm provides an integration named database
with the charm-db
interface, and similarly the web app charm requires an integration named database
with the charm-db
interface.
Note that the distinction between interface name and integration name enables developers to specify a name for the integration that makes the most sense in the context of their charm (which may be different from the name provided by the charm on the other end of the relation), while ensuring that the two are compatible where the interface name is the same. Using the above example, the database charm author could replace the name of the integration with charm-db
, but the application charm would still be compatible, as the interface
names remain the same.
Peer integrations
Charms can declare peer integrations, which causes each unit of an application to respond to the other units deployed in the same application. A peer integration is defined in exactly the same way as any other integration. For example, our fictional database could define a replicas
integration, with the interface charm-db-replica
peers:
replicas:
interface: charm-db-replica
Peering integrations are particularly useful when your application supports clustering. Consider the implications of operating applications such as MongoDB, PostgreSQL, and ElasticSearch where clusters must exchange information amongst one another to perform proper clustering. Peer integration can be useful for coordinating operations against a clustered application without downtime – for example upgrading package versions or applying new configuration. Additionally, forming a peer integration is the only way to obtain a reliable IP address from other pods deployed for the same application, though this may be less relevant depending on the underlying substrate. This information is populated automatically for each unit using an implicit integration.
Integration and interface naming
The integration namespace is unrestricted with the exception that you may not provide an integration named juju
, nor have its name begin with juju-
. Charms attempting to provide integrations in this namespace will trigger an error.
Interface names are strings that must only contain characters a-z
and -
. Names must not start with -
. The interface name is the only means of establishing whether two charms are compatible for relation; and carries with it nothing more than a mutual promise that the provider and requirer somehow know the communication protocol implied by the name.
Implicit integrations
Implicit integrations allow for simple integrations to be formed without requiring any modifications to target charms. Implicit integrations exist in the reserved namespace. There is currently only one such integration provided to all deployed applications: juju-info
.
The juju-info
integration captures very select data from the remote unit:
private-address
public-address
A common use for this type of relation is when authoring a subordinate charm that can be integrated with any other charm – in this case the juju-info
implicit integration can be used. For example, the rsyslog-forwarder
charm is a subordinate charm that requires a valid scope: container
integration named logging
. In the event that the principal charm doesn’t provide this, the logging charm author can use juju-info
:
requires:
logging:
interface: logging-directory
scope: container
juju-info:
interface: juju-info
scope: container
The administrator can then issue the following command:
juju integrate some-app rsyslog-forwarder
If the some-app
charm author doesn’t define the logging-directory
interface which would configure the principal charm to log files into a directory, Juju will use the less-specific juju-info
interface to create a config that configures the principal charm to forward syslog to the IP of the integrated application (using the information available from the juju-info
implicit integration).
Integration events
Defining integrations is only the first step; for integrations to be useful, charm developers must implement handlers for the various events triggered by integrations, and ensure they implement the advertised interface.
Event types
The CharmBase
class automatically defines 5 events per specified integration (where <name>
is the name of the relation).
Event name | Event Type | Description |
---|---|---|
<name>_relation_created |
RelationCreatedEvent |
Triggered when a new integration is created. This can occur before applications have started. |
<name>_relation_joined |
RelationJoinedEvent |
Triggered when a new unit joins a relation. This event only fires when the remote unit is first observed by the unit. Callback methods bound to this event may set any settings that can be determined by only using the joining unit’s name and the remote private-address setting. |
<name>_relation_changed |
RelationChangedEvent |
Triggered whenever there is a change to the integration data. This event is triggered whenever there is a change to the data bucket for an integrated application or unit. Look at event.relation.data[event.unit] or event.relation.data[event.app] to see the new information, where event is the event object passed to the callback method.This event always fires once, after RelationJoinedEvent , and will subsequently fire whenever that remote unit changes its settings for the relation.Callback methods bound to this event should be the only ones that rely on remote relation settings. They should not error if the settings are incomplete, since it can be guaranteed that when the remote unit or application changes its settings, the event will fire again. The settings that may be queried, or set, are determined by the relation’s interface. |
<name>_relation_departed |
RelationDepartedEvent |
Triggered when a unit leaves a relation. This is the inverse of the RelationJoinedEvent , representing when a unit is leaving the relation (the unit is being removed, the app is being removed, the relation is being removed). It is fired once for each unit that is leaving.Callback methods bound to this event may be used to remove all references to the departing remote unit, because there’s no guarantee that it’s still part of the system; it’s perfectly probable (although not guaranteed) that the system running that unit has already shut down. Once all callback methods bound to this event have been run for such a relation, the unit agent will fire the RelationBrokenEvent . |
<name>_relation_broken |
RelationBrokenEvent |
Triggered when a relation is removed. If an integration is being removed ( juju remove-relation or juju remove-application ), once all the units have been removed, this event will fire to signal that the relationship has been fully terminated.The event indicates that the current integration is no longer valid, and that the charm’s software must be configured as though the relation had never existed. It will only be called after every callback method bound to RelationDepartedEvent has been run. If a callback method bound to this event is being executed, it is guaranteed that no remote units are currently known locally. |
Integration data (‘databags’)
The primary means for applications to communicate over an integration is using integration data. This is exposed to the charm as a property of the event passed to the relevant callbacks. All integration events inherit from the RelationEvent
class, which encapsulates the relevant Relation
at <event>.relation
. The Relation
includes a data
parameter of type RelationData
which is a mapping containing all relation data stored by participating applications and units.
Integration data comprises a number of “databags” (or “data buckets”). Each application and each unit has its own databag, which is a Python str:str
mapping. There are some rules about accessing integration data:
- Units can read and write their own unit databags (e.g.
event.relation.data[self.unit]
) - Units can read from any other unit databags (e.g.
event.relation.data[event.unit]
) - Application leaders can read and write their application databags (e.g.
event.relation.data[self.app]
) - Non-leaders can read from other application databags (e.g.
event.relation.data[event.app]
)
A visualization:
Data stored in a RelationData
mapping must be of type string
.
Integration data size is limited to 16MB for the entire unit (all keys and values together), but individual data update operations are limited by subprocess
command-line max length, ARG_MAX
(which may be as low as 4096). This ARG_MAX limitation will be removed in ops 1.6 and greater.
Integration data is persisted to the controller database, and therefore persists across invocations of the charm code and through the various event handlers that may access it.
The framework also provides a mechanism for a charm author to fetch an integration from the model by name or ID. This can be useful in cases where methods (that are not integration event callbacks) need to access integration data. An example of this might be updating application data on a peer relation in the leader-elected
event callback. Details of this method are documented in the API docs, and it is used in context in the examples provided. Where there are multiple integrations present for a single relation name, the get_relation
method takes an id
of the integration to fetch. In these cases it may be easier to iterate over the relations
property of the model, which returns a Mapping
of all integrations.
When implementing a more complex integration, it might be useful to codify your integration by providing a charm library that provides helper classes for developers who need to consume your integration.
Examples
Requires/provides integration
Here we will construct a trivial example of an integration to bring these concepts to life.
For a given metadata.yaml
:
requires:
demo:
interface: demo
We could implement the following callback for the demo_relation_changed
event:
# ...
class SpecialCharm(ops.CharmBase):
# Set up some stored state in a private class variable
_stored = ops.StoredState()
def __init__(self, *args):
super().__init__(*args)
# ...
# Relation handling
self.framework.observe(self.on.demo_relation_changed, self._on_demo_relation_changed)
self.framework.observe(self.on.demo_relation_broken, self._on_demo_relation_broken)
# Initialise our stored state
self._stored.set_default(apps=dict())
def _on_demo_relation_changed(self, event: ops.RelationChangedEvent) -> None:
# Do nothing if we're not the leader
if not self.unit.is_leader():
return
# Check if the remote unit has set the 'leader-uuid' field in the
# application data bucket
leader_uuid = event.relation.data[event.app].get("leader-uuid")
# Store some data from the relation in local state
self._stored.apps.update({event.relation.id: {"leader_uuid": leader_uuid}})
# Fetch data from the unit data bag if available
if event.unit:
unit_field = event.relation.data[event.unit].get("special-field")
logger.info("Got data in the unit bag from the relation: %s", unit_field)
# Set some application data for the remote application
# to consume. We can do this because we're the leader
event.relation.data[self.app].update({"token": f"{uuid4()}"})
# Do something
self._on_config_changed(event)
def _on_demo_relation_broken(self, event: ops.RelationBrokenEvent) -> None:
# Remove the unit data from local state
self._stored.apps.pop(event.relation.id, None)
# Do something
self._on_config_changed(event)
# ...
From the example above, note:
- The callback will only run if the current unit is the application leader.
- The callback gets some application data from the remote unit, and stores it locally in state.
- It fetches unit data from the remote unit that joined the relation.
- It sets some application data for itself, to be consumed by the remote application.
A corresponding application that could relate to this could specify the following metadata.yaml
:
provides:
demo:
interface: demo
And the following integration event callback which:
- Sets some application data for the local application, if the unit is the leader
- Sets some unit data, containing just the unit’s name
- Fetches some application data from the remote application and stores it in state
def _on_demo_relation_changed(self, event: ops.RelationChangedEvent) -> None:
# If we're the current leader
if self.unit.is_leader():
# Set a field in the application data bucket
event.relation.data[self.app].update({"leader-uuid": self._stored.uuid})
# Set a field in the unit data bucket
event.relation.data[self.unit].update({"special-field": self.unit.name})
# Log if we've received data over the relation
if self._stored.token == "":
logger.info("Got a new token from '%s'", event.app.name)
# Get some info from the relation and store it in state
self._stored.token = event.relation.data[event.app].get("token")
Peer integration
This example illustrates a trivial peer integration, where the events can be observed using juju debug-log
. The logging from the charm will demonstrate how each charm is notified of peers leaving/joining, and how the relation data is eventually consistent between units.
The basic premise behind this example is a simple clustered application that needs to present a single “master IP address” to another relation. Using the peer integration and the concept of application leadership, the IP address of the current leader is stored in the application data bucket on the peer relation, and cached locally into state on each of the followers. If a new leader is elected, the peer integration is fetched from the model and the application data stored on the peer relation is updated – triggering each of the other peers to update their local state.
The metadata.yaml
for the example below contains the following:
peers:
replicas:
interface: charm-replica
And in the charm code:
# ...
class DemoCharm(ops.CharmBase):
"""Charm the service."""
_stored = ops.StoredState()
def __init__(self, *args):
super().__init__(*args)
# ...
self.framework.observe(self.on.leader_elected, self._on_leader_elected)
self.framework.observe(self.on.replicas_relation_joined, self._on_replicas_relation_joined)
self.framework.observe(self.on.replicas_relation_departed, self._on_replicas_relation_departed)
self.framework.observe(self.on.replicas_relation_changed, self._on_replicas_relation_changed)
self._stored.set_default(leader_ip="")
# ...
def _on_leader_elected(self, event: ops.LeaderElectedEvent) -> None:
"""Handle the leader-elected event"""
logging.debug("Leader %s setting some data!", self.unit.name)
# Get the peer relation object
peer_relation = self.model.get_relation("replicas")
# Get the bind address from the juju model
# Convert to string as relation data must always be a string
ip = str(self.model.get_binding(peer_relation).network.bind_address)
# Update some data to trigger a replicas_relation_changed event
peer_relation.data[self.app].update({"leader-ip": ip})
def _on_replicas_relation_joined(self, event: ops.RelationJoinedEvent) -> None:
"""Handle relation-joined event for the replicas relation"""
logger.debug("Hello from %s to %s", self.unit.name, event.unit.name)
# Check if we're the leader
if self.unit.is_leader():
# Get the bind address from the juju model
ip = str(self.model.get_binding(event.relation).network.bind_address)
logging.debug("Leader %s setting some data!", self.unit.name)
event.relation.data[self.app].update({"leader-ip": ip})
# Update our unit data bucket in the relation
event.relation.data[self.unit].update({"unit-data": self.unit.name})
def _on_replicas_relation_departed(self, event: ops.RelationDepartedEvent) -> None:
"""Handle relation-departed event for the replicas relation"""
logger.debug("Goodbye from %s to %s", self.unit.name, event.unit.name)
def _on_replicas_relation_changed(self, event: ops.RelationChangedEvent) -> None:
"""Handle relation-changed event for the replicas relation"""
logging.debug("Unit %s can see the following data: %s", self.unit.name, event.relation.data.keys())
# Fetch an item from the application data bucket
leader_ip_value = event.relation.data[self.app].get("leader-ip")
# Store the latest copy locally in our state store
if leader_ip_value and leader_ip_value != self._stored.leader_ip:
self._stored.leader_ip = leader_ip_value
if __name__ == "__main__":
ops.main(DemoCharm)
# ...
To illustrate how these events play out, open a separate terminal and run juju debug-log
. Use the juju
CLI to add units to the charm, observing the events in the debug log, then remove them and see the reverse.
Handling integration data from non-relation hooks
Sometimes you have to manipulate integration data in a context where there is no “current” integration; in other words, when a non-relation-related event occurs. For example, when a config-changed
event occurs. Suppose the charm has the same demo
interface as above.
def _on_config_changed(self, event: ops.ConfigChangedEvent):
# from the Model, you can access all relations your charm has at the moment,
# by endpoint name.
my_relations: Mapping[str, List[Relation]] = self.model.relations
# the list of relations bound to this charm's `demo` endpoint.
demo_relations: List[Relation] = my_relations['demo']
for relation in demo_relations:
# this is how you get all of the databags in this relation.
# RelationData is a mapping from the "owner" of a databag to the databag's contents..
data: ops.RelationData = relation.data
# LOCAL ENTITIES:
# so to get the app databag of **this** application, you can:
this_app_data: ops.RelationDataContents = data[self.app]
# and to get the unit databag of **this** unit:
this_unit_data: ops.RelationDataContents = data[self.unit]
# we can get a list of our peer units like so:
peers: List[Unit] = [u for u in relation.units if u.app is self.app]
# and now we can get to the unit databags of the other units of **this** app:
peer_databags: List[ops.RelationDataContents] = [relation.data[peer] for peer in peers]
# REMOTE ENTITIES:
# to get the app databag of this relation's remote side:
remote_app_data: ops.RelationDataContents = relation.data[relation.app]
# to get a list of the remote units:
remote_units: List[ops.Unit] = [u for u in relation.units if u.app is not self.app]
# and now we can get to the unit databags of the other **remote** units:
remote_unit_databags: List[ops.RelationDataContents] = [relation.data[rem] for rem in remote_units]
Testing integrations
Integrations are fundamental to Juju’s model-driven approach, and therefore they should be tested rigorously to ensure they behave as expected, and that charms present the right interfaces to other charms.
Harness provides a number of methods to aid testing relations, which are demonstrated below:
# ...
harness = ops.testing.Harness(MyCharm)
# Do initial setup here - note that adding a relation does not trigger any events
# in the context of Harness. While this differs from Juju, it makes testing simpler
# It takes the relation name, and remote application as arguments
relation_id = harness.add_relation('db', 'postgresql')
# This will trigger a relation-joined event, it takes the relation id,
# and name of the unit to join to the relation
harness.add_relation_unit(relation_id, 'postgresql/0')
# This method both updates the relation data, and triggers a relation-changed event
# This takes the relation id, name of the application or unit, and a value
harness.update_relation_data(relation_id, 'postgresql/0', {'key': 'val'})
harness.set_leader(True)
harness.update_config({'initial': 'config'})
# Trigger the same events Juju would when deploying/installing charm.
# Specifically here: install, db-relation-created('postgresql'), leader-elected, config-changed, start, db-relation-joined('postgresql/0'), db-relation-changed('postgresql/0')
harness.begin_with_initial_hooks()
# ...
The final method regarding integration testing is get_relation_data()
which returns the relation data bucket for a given application or unit. Example usage:
# ...
self.assertEqual(self.harness.get_relation_data(relation_id, "postgresql/0"), {
"hostname": "dbhost",
"port": "4444",
"user": "charmuser",
"pass": "supersecret"
})
# ...
Note that when calling get_relation_data
or update_relation_data
you are able to read/write relation data for both the charm being tested, but also the ‘remote’ charm in the relation. This is essential to ensure that your charm responds to remote changes in relation data correctly, though can become difficult where interfaces are complicated or subject to change (perhaps during development).
In both cases, the get_relation_data
and update_relation_data
take an argument to specify which charm is having it’s data read or changed – even though one or more of those charms may be entirely simulated due to the nature of Harness.
Last updated 22 days ago.