Model-driven operations

Application management is about business decisions

In a large organisation, it is important to separate out business intention from technical implementation. For example, if we are going to set up a service, we might run it in a staging environment, and we might set up test environments, as well as running it in production.

In each case, we take different business decisions about resource allocation, even though the service is broadly the same. The business decisions that are important are:

  • Where should the application run? For example, it might run on a public cloud, or a private cloud, a Kubernetes cluster on one of those, or it might be split across several different substrates. In each case, it is a business decision, driven by cost, privacy, compliance and latency, where we run the applications. We might test on a public cloud, and run production on private infrastructure, or the other way around. The applications are the same, the integration is the same, the operations are the same, the business decision is about locality.

  • Which software components should be included in the scenario? Any significant service will require multiple software components, or applications, that are integrated and working together to create the service. The business decision here is choice of component and version. We might want to use a specific version of MySQL. Or we might want to use Amazon RDS for a managed MySQL. We could of course use our own short-lived MySQL instance for testing, and RDS for production, or the other way around.

  • What capacity should be allocated to the scenario? Compute and storage are expensive. The same software can be run with fewer resources, or more, depending on the business need. For a test or staging environment, we might allocate small machines on a cloud, for a production environment we might allocate large bare metal servers on premise. The technical details of operating the applications should accommodate those business needs.

  • How should that scenario be integrated into the wider estate? While the applications in the service will be tightly integrated, there may also be integration work with systems that are not strictly part of this service. For example, if I have an institution-wide intrusion detection system, I may want to wire production and staging scenarios up to that, but not CI/CD test run scenarios. Similarly, I may not want to send all the logs from all my test runs to Splunk, even if I use that SAAS for my staging and production environments.

This is application management, not configuration management. Making these decisions is about expressing the business intention with regard to the scenarios. With Juju, we capture this business intention explicitly. Instead of focusing on the details of configuration files, and translating the business decisions to a large amount of configuration management code, we encode the business decision itself.

It is almost as if a traditional operations engineer ‘compiles business intentions’ into configuration management code, while Juju preserves the source code of the business intention, translating it at runtime to the particular environment where the scenario is playing out. Juju works at a higher level, which makes the resulting artifact much easier to understand and share.

Scenarios, models and substrates

We’ll use the terms ‘substrate’, ‘scenario’ and ‘model’ a lot. We need all of those terms because large services are often built up of many components spread across several different places.

For example, we might have a database on some very secure hardware, like a mainframe. We might also have bare-metal machine learning happening on some adjacent commodity servers with GPU acceleration, and the core of the application running on VMware. The front-end might be running on Kubernetes, and we might be shipping data off to a public cloud for data warehousing in a SAAS offering there.

A substrate is a place where applications can run. In the example, VMware is a substrate. The bare metal servers are a different substrate. The mainframe is a substrate. Public cloud is a substrate. And the Kubernetes cluster hosting the front-end is also a separate substrate. So this scenario has five different substrates, five different places where we want to drive software.

Broadly speaking, there are machine substrates and container substrates.

A machine substrate is a place where you can get physical or virtual machines. VMware gives you virtual machines, as does any public cloud. Bare metal is also a machine substrate; those machines can be allocated manually, or from an on-demand bare metal cloud system like MAAS.

Kubernetes is a container substrate. You won’t get machines there, you run Docker images in containers. Any Kubernetes should work, whether it is a public cloud offering like AKS, GKE or EKS, or a Kubernetes that you have deployed yourself.

A model is a canvas on a particular substrate. The model is used to group applications that are being operated together for a common purpose on a common substrate. The model will capture the applications, their integration, configuration, and resource allocation. Since each model is on a single substrate, and the service as a whole may span multiple clouds/k8s-clusters, it may require several models to provide the canvases for all the different applications in the service.

A scenario is the whole picture of all the models containing all the applications working together to provide the service. The scenario may be spread across several models, each of which enables the placement of applications on a particular substrate. A scenario will typically be replicated in dev, staging and production. The scenario can span clouds and data centres, it can span Kubernetes and machine workloads.

So, applications sit in models, models sit on substrates, and scenarios are made up of one or more models so that they can span multiple substrates.

Model-driven operations capture business intent

The model doesn’t just represent a substrate conceptually. It captures actual resource allocations too. For example, a model on a public cloud keeps track of the number and type of virtual machine instances allocated to the model. Networks are declared in a model too, as are classes of storage. This allows us to place an application in the model, bind aspects of it to particular networks, and have it consume a certain amount of a particular class of storage.

Unlike a traditional management system, model-driven operations understand why these resources were allocated and how they’re being used. A future administrator looking at a model understands more clearly the intention of the people setting it up. The model reflects application management decisions directly.

Where we run the applications

The first business decision is the substrate for the model. It can be any public or private cloud IAAS, including bare metal machines, or it can be on any Kubernetes cluster. The model starts as a blank canvas. Applications will be placed on the canvas. By creating the canvas on a particular substrate, we are deciding where the applications will run.

A single scenario might in fact involve several models on different substrates, because it involves applications that must run in multiple different clouds. I might have some components on my mainframe, some on a public cloud, some on a different cloud Kubernetes cluster, and some on servers. Each of those compute substrates would get a model, and applications placed in each model will run on that substrate.

Who is responsible for each application

Models also allow different groups of administrators to be responsible for particular subsets of the applications in the scenario. Since permissions are allocated by model, it is convenient to represent organisational boundaries in models as well. One team that has responsibility for the data lake could have permissions to administer a model with those components, another team that is responsible for machine learning could have permission to administer a second model.

When we have separate teams responsible for subsets of the applications, we create a scenario with two models on the same substrate to segregate responsibilities. Since applications can be integrated across model boundaries, multiple models do not complicate the overall scenario design. So the second business decision is who has permission to administer each model.

Which applications we compose in the scenario

The third business decision is which applications will be needed in that scenario. Operators are placed into models using the Juju operator lifecycle manager. The operators are integrated through matching integration points. Each line of integration represents a particular way in which those two operators are composed. Operators can be composed in many ways, if two operators have multiple pairs of matching integration points then there can be multiple lines of integration between them. The net effect is a rich application graph on the canvas.

This graph is an abstraction. The same graph could describe a small deployment, on Raspberry Pi hosts, or a large deployment, on dual-socket x86 servers. The graph describes the logical relationships between the applications. We can reuse this graph on different substrates to build multiple copies of the same idea, for example development, staging and production.

How much we spend on capacity

To make this abstract model real, we need to provide some compute capacity, some storage, and possibly some networks on which those applications will run. This is the fourth business decision that we must take — how much we want to spend on capacity for this particular scenario.

On a machine substrate, like IAAS or VMware or bare metal with MAAS, compute capacity takes the form of machines. By allocating machines to the model we make a business decision — how much hardware we want to spend on this application. In the cloud, compute capacity is explicitly financial because it is clear that you are spending money for each instance hour. Bare metal has the same financial considerations. Every host has a cost, placing more hosts into the model, or more expensive hosts is a business decision.

The same is true of storage — by deciding which kinds of storage to provide, and how much, we are making business decisions about financial resource allocation to the application that are captured in the models.

Storage is described as a set of storage classes, typically mapped to substrate-specific storage types. On AWS, one can construct a storage class for the model out of EBS. Different storage classes would have different performance, durability, reliability and costs. Again, the model records these business decisions explicitly. Dev, staging and production scenarios might each use exactly the same application configuration and integration, but use different storage classes, tuned for speed, reliability, or cost.

How we integrate with the wider estate

The fifth and final business decision is wider integration. What other systems, outside this particular scenario, are important to integrate?

It is very common to have central logging, monitoring and alerting. It is also common to use shared resources for things like databases in organisations which have specialist practitioners that run such systems. Those central systems are not specific to the scenario, but the scenario will need to be integrated with them.

We can draw lines of integration across models. We use this both to integrate applications that must run on different substrates in the same scenario (because each model is on a single substrate). We can also use this to integrate applications across scenarios, or across boundaries in the organisation.

If we have a set of models with large databases in them, operated securely by our DBA team, then we might use a cross-model integration to those for the data in our production scenario. But dev and test scenarios might not use those central databases, instead using a local copy of the database provided by that database operator, or a SAAS database like Amazon RDS.

Permissions

The model serves as a useful boundary for permissions. User and group permissions are set at the model level with role-based access controls (RBAC).

So models group sets of tightly integrated applications on the same substrate, which are managed by the same group of people. It is very common to see organisational boundaries reflected in the sets of models managed by the organisation. The DBA group will have a set of models with databases. The data lake team will have a few models which represent the large-scale big data resources in the company. Different teams of DevOps will have models of the applications they are driving.

Cross-model integration requires both models to agree, so both teams need to agree to integrate applications that they respectively control, as it should be.

Permissions can be granted and revoked quickly, so it is common to see teams granting short-term permissions to a person or team in order to collaborate, after which the permissions are dropped to revert to the more independent operating norm.

Applications, workloads and operators

It is easy to talk about operators and applications as though they are the same thing, but they are not.

The operator is a separate piece of code from what most people consider to be the application. The MySQL Operator is not MySQL, it is the code-which-installs-and-configures-MySQL. The operator, packaged as a charm, is deployed by the operator lifecycle manager, and in turn stands up its workload, which is what most people call the application.

We use the term workload when we want to be specific about the ‘software being driven by the operator’. A single operator might drive several workloads in the same container or on the same machine; for example, a database and transaction monitor which is part of the same application.

In the model, the application is a name only. We ‘deploy an application’ into the model. If the operator was not provided locally then Juju finds the right operator for that application by looking for it in Charmhub, and then Juju instantiates the operator on the substrate. For a machine substrate, Juju will typically get a fresh machine, install the lifecycle manager agent, and then install the operator.

An application in the model can be configured. This configuration is high-level configuration, it is not the explicit underlying workload configuration, but rather a YAML abstraction which the operator will map down to actual configuration files if needed. Three different MySQL applications in the model can have three different configurations, allowing one to be tuned for performance, another for resilience, and the third for read-only usage, for example.

It is possible to deploy the same operator multiple times in the same model, under different application names. For example, I might have three different databases in the same model, so I could use the MySQL operator three times with different application names. When integrating operators, we use their application names in the model, so there is no confusion at the level of the operators themselves. Three separate machines or containers will be instantiated, three separate copies of the MySQL operator will be deployed, they can be configured differently and they will respond to events completely independently.

So ‘application’ is the name in the application graph in the model, an application in the model is defined by its operator and configuration, an operator can be used multiple times in a model under different names. The workload is the actual processes driven by the operator when it is running.

Subordinate applications

There are certain software components which play a supporting role. They are essential, but they are not the reason for the model to exist. In business terms, they are not the drivers of resource allocation.

For example, consider a heartbeat monitor for a high-availability database. The reason to spend the resources is to run the database, not the monitor. We are allocating machines and memory and disk to the database. But we want to make sure that the monitor is there, because it helps deliver high availability.

The Juju OLM introduces the notion of subordinate applications which are attached to a primary application in the model. Wherever we instantiate the primary application, we will also instantiate the subordinate application.

In the example, the heartbeat monitor is a subordinate application to the database application. If we scale out the database, we will exactly follow with the heartbeat monitor, ensuring that the heartbeat monitor can play its supporting role everywhere it is needed.

Units represent application scale

Many applications can deploy parallel instances for resilience, throughput or both. Mission-critical enterprise applications often work in a failover mode for high availability, or designate some instances as read-only for scale. With newer cloud-native software, where ‘scale- out’ and consensus algorithms such as RAFT and PAXOS have become popular approaches to capacity and availability, parallel instances typically offer robust resilience with automatic cutover, or the ability to scale and process large amounts of data in parallel through sharding.

So an ‘application’ may, in fact, be multiple instances of the same software component, running in parallel, and configured to coordinate in a cluster between themselves either directly or with the help of additional components. For example, a heartbeat solution like Pacemaker might be monitoring two instances of a database, and controlling the cutover from standby to primary. In this case, the database would have two instances, and both would be integrated with instances of Pacemaker.

The Juju OLM keeps track of these separate instances of the software, calling them ‘units’ of the application in the model. Units are parallel instances of the software making up the application. So, when we put a MongoDB application into a model, we can specify that it should have a particular number of units. By default, an application will only have a single unit, mapping to a single instance. Specifying two or three units means that there will be two or three parallel instances of the software running, each of which will be deployed and managed by the operator of that application.

In general, there will actually be parallel operators for these parallel instances, too.

Imagine we want to spin up a highly available MongoDB cluster of three bare metal machine instances. We would create a model, allocate the three bare metal machines to the model, create a MongoDB application (with the MongoDB operator) and set it to have three units.

At this stage, the OLM will place three separate copies of the operator onto the three different machines. Those separate operators will each install their instance (unit) of MongoDB on their machine. However, those operators will know that they are instances of the same MongoDB application in the model, and so instead of providing three separate MongoDB databases, they will configure their respective instances to cluster into a single highly available database.

It is important to remember, then, that a given instance of an operator is typically looking after only one instance of the application software. When there are multiple units in the application, there will be multiple parallel instances of the application software, controlled by multiple parallel instances of the operator. That means that events could be handled by separate instances of the operator at the same time.

These operators are themselves a distributed system, and it is well known that distributed systems can be challenging to design, develop and debug. For this reason, the Juju OLM provides a set of services to operators that greatly simplify development of high quality operators for mission-critical systems. The operators can use the OLM to coordinate, prioritize, elect a leader, and provide persistence, solving many of the key challenges in distributed systems design.

So, in a sense, each operator is acting on one unit, rather than the whole application. There is no ‘operator for the application’, there are only ‘unit operators’. With the three instances of MongoDB, it is not possible to point at any one of them and say ‘the database is there’ because the database cluster is actually spread across all three instances equally. Similarly, it is not possible to point at a single operator and say ‘the application in Juju is there’, because all three operators play their part in running the conceptual application in the model.

In distributed systems, there is the notion of a ‘leader’ in a cluster. The Juju OLM provides leader election as a service to operators. This means that one of the operators is naturally in a position to ‘speak for the application’, and in fact, that operator can update information in the lifecycle manager that pertains to the application as a whole.

There is of course one exception to the rule that parallel application instances get parallel operator instances. On Kubernetes, it is possible to create an operator which runs in its own ‘pod’, separate from the application. In this case, increasing the number of units in the application will increase the number of workload pods in the Kubernetes cluster, but not affect the number of operator pods at all. The single operator pod controls all the workload pods using standard capabilities in Kubernetes.

This rather simplistic approach is common in Kubernetes implementations of the operator pattern. It suffers from some limitations, such as ensuring high availability of the operator itself, and limited ability to interact closely with the application it is controlling. That is why the Juju OLM also supports more fine-grained operator placement with operators as sidecars, in which case there will be one operator per unit in the application, and they will have more direct control of their application instances.

A further consideration is subordinate applications. The scale of a subordinate application follows its primary application. If we deploy a database and a subordinate heartbeat monitor, the scale of both is determined by the scale of the database. If we add three units to the database, we will automatically add three more units to the heartbeat monitor, on exactly the same machines.

Resources

It is common for an operator to require some large pieces of content. For example, the operator may need assets for a game, or it may need a CD image from which the application installation proceeds, or it may need a neural network model to be provided.

With the Juju OLM, resources can be declared in the operator package (“charm”) metadata. Such resources can be retrieved by the operator on demand, and updated independently of the operator code itself. This allows new versions of the resource to be propagated to deployments without having to ship new versions of the application or the operator itself.

Integration points and the application graph

A key feature of the Juju OLM is its support for declarative integration between operators. We can deploy two operators with the lifecycle manager, then we can request that they integrate, and automatically have the applications reconfigure themselves to work with one another. We do not need to write any custom integration code in order to integrate two operators that use the Juju OLM, because integration is a first-class design goal of Juju.

Of course, the two operators must already have the necessary integration code embedded in them in order to complete that automatic integration code. The operator metadata, in the charm package, describes a set of typed, directional integration points. If two operators in a model have integration points of the same type, and opposite directions, then it is possible to integrate them.

Think of the ‘direction’ of the integration point like the pole of a magnet. Two north poles repel one another, but a north and a south pole attract one another. We can integrate two operators which have integration points that ‘match’ in this way — the same type and opposite directions.

When we integrate two operators, we get a line between two integration points on the model.

As we integrate multiple operators, the model turns into an application graph. Usually, every application in the model will be related to one or more other applications in the model.

This ability to compose operators into a bigger picture allows the individual operators to be cleaner, simpler and more reusable. They do one thing well, which is a key philosophy in effective systems design.

Sharing and reusing scenario definition bundles

A model is a very valuable summary of business intention. It describes very succinctly, which apps, on which substrates, with which administrators, with what high level configuration and integration, and at what scale over which resources. It neatly captures everything you need to deliver that business intention repeatedly: just redeploy exactly the same model, and you get exactly the same service.

It makes sense, then, to be able to reuse whole scenarios.

The Juju OLM can ‘export’ a model to a text file in YAML format, which can easily be ‘imported’ into a model somewhere else, recreating the applications, scale, configuration and integration pattern perfectly. A Juju bundle is a complete description of multiple applications which can be shared, reused, and put into version control.

Bundles are often used as a CI/CD primitive, or for team control of a desired application topology.

Bundles can also be published in Charmhub to allow other people to get exactly the same scenario just by standing up a clean model and deploying the bundle into it. So, for example, there are existing bundles that describe Kubernetes clusters, OpenStack private clouds, Hadoop deployments, and many other complex topologies of software.