Linkage in Grafana

Traces make up just one of the three pillars of observability.

• Metrics are generally the What is going wrong, e.g. A high rate of error logs or non-2xx response status codes.
• Traces are the Where, e.g. requests between certain services taking a long time or certain trace paths often resulting in errors.
• Logs are the Why, they contain messages describing the errors and why they occurred.

With this in mind, the ability to quickly correlate the three provides a significant reduction in the amount of time required to identify and diagnose issues.

Logs to Traces

Logs can be linked to their corresponding traces via a trace ID, providing that the log includes one as a field.

Loki settings allowing linkage to Tempo spans via a trace ID field.

Traces to Logs

Traces can be linked to potentially relevant logs by matching a span's attributes to a log's labels. For example, transaction logs (request.log, transaction.log and fr-exception.log) are logged with an app_name label. You can use this to show logs that occurred within a specific application during a span's duration.

Tempo settings allowing linkage to Loki via an app_name tag.

Application Name Inconsistency

Sometimes logs occur at times when a transaction's Application Name is not known and the log is tagged with the default Application Name, MyApplication. This is typically the 'Executing' log message in request.log.