How to instrument your charm with tracing telemetry

In order to instrument a charm with tracing telemetry, you will need to:

1. Set up a model with cos-lite:

juju add-model clite
juju deploy cos-lite --trust

See more: Charmhub | cos-lite

2. Deploy tempo:

juju deploy tempo-k8s

See more: Charmhub | tempo-k8s

3. Integrate tempo with cos-lite:

jhack imatrix fill

See more: jhack, GitHub | jhack > imatrix > fill

Alternatively, you can integrate manually by:

juju integrate tempo:logging loki:logging
juju integrate tempo:ingress traefik:ingress
juju integrate grafana:grafana-source tempo:grafana-source
juju integrate prometheus:metrics-endpoint, tempo:metrics-endpoint
juju integrate grafana:grafana-dashboard, tempo:grafana-dashboard
juju integrate traefik:tracing, tempo:tracing
juju integrate prometheus:tracing, tempo:tracing

At this point you should have a working cos-lite plus tempo deployment; something like this:

4. Fetch the charm_tracing and the tracing libs:

charmcraft fetch-lib charms.tempo-k8s.v1.charm_tracing
charmcraft fetch-lib charms.tempo-k8s.v2.tracing

See more: charmcraft fetch-lib, Charmhub | tempo-k8s > Libraries > charm-tracing, Charmhub | tempo-k8s > Libraries > tracing

5. Add opentelemetry exporter dependency used by charm_tracing to your requirements.txt:

opentelemetry-exporter-otlp-proto-http>=1.21.0

6. Add an integration with tempo:

# in charmcraft.yaml
provides:
  tracing:
    interface: tracing
    limit: 1

See more: File charmcraft.yaml > provides

7. Instrument your charm code:

# in /your/charm/project/src/charm.py
from charms.tempo_k8s.v1.charm_tracing import trace_charm
from charms.tempo_k8s.v2.tracing import TracingEndpointRequirer

@trace_charm(tracing_endpoint="tracing_endpoint")
class MyCharm(CharmBase):
    def __init__(self, *args):
        super().__init__(*args)
        # add a provider wrapper for the tracing endpoint
        self.tracing = TracingEndpointRequirer(self, protocols=["otlp_http"])

    @property
    def tracing_endpoint(self) -> Optional[str]:
        """Tempo endpoint for charm tracing."""
        if self.tracing.is_ready():
            return self.tracing.get_endpoint("otlp_http")
        return None

8. Pack, deploy, and integrate your charm with cos-lite:

charmcraft pack
juju deploy ./my-charm-operator_ubuntu-20.04-amd64.charm mycharm \
  $(yq eval '.resources | to_entries | map(select(.value.upstream-source != null) | "--resource " + .key + "=" + .value.upstream-source) | join(" ")' charmcraft.yaml)
juju integrate mycharm:tracing tempo-k8s:tracing

See more: charmcraft pack, Juju | juju deploy, Juju | juju integrate

9. View the traces for the charm.

Open the Grafana dashboard in a browser (see here for more detailed instructions).

Next, navigate to the traces for your charm:

• go to Explore and select the Tempo datasource.
• pick the service_name you gave to MyCharm above (the default is the application name) to see the traces for that charm
  
• click on a trace ID to visualize it. For example, this is the trace for an update-status event on the Tempo charm itself:
  

Mapping events to traces with jhack tail

jhack tail supports a -t option to show the trace IDs associated with a charm execution:

This means that you can tail a charm, grab the trace id from tail, put it in the grafana dashboard query and get to the trace in no time.