Skip to content

Explore

Explore allows you to query and analyse all metrics, logs and traces ingested to your cloud account.

Cloud Explorer from FusionReactorAPM on Vimeo.

You can create new data views and filter data in any way you require.

Warning

While with explore you can technically search for any amount of data within your cloud account, there are strict execution times on queries that will restrict performance heavy queries.

Datasource Selection

Within Explore you will have 3 datasource's configures, which you can toggle between using the selector.

  • Metrics will contain any metric sent from a FusionReactor agent, plus some additional metrics created within our ingest engine.
  • Traces will contain any slow or error transactions sent from a FusionReactor agent, requests viewed in the recent or running tabs of the server view will not appear here as they are not ingested.
  • Logs will contain any logs sent to FusionReactor, either from a FusionReactor agent or log shipper.

Note

Going forward we intend to allow ingest for traces and metrics from a non FusionReactor agents, however at this time this feature is not available.

Metric Queries

To explore metrics, you use the metric browser.

Within the metric browser you can:

  • Select one or more metrics to view
  • Filter metrics by any label, such as instance, group or application
  • Execute functions such as sum, count, or avg on metrics
  • Compare metrics by using mathematical functions
  • Combine multiple queries in a single view
  • View data as a various chart types, or as raw data in a table.

The metrics browser will create data views for you using promql.

Using Promql directly you can create powerful and complex queries to get the most out of your FusionReactor data.

Trace Queries

To explore traces we advise using the search feature.

Within search, you can specify:

  • service name - Instance from which the trace originates
  • span name - The url or action of a transaction
  • tag - A label applied to a trace, for example txnId or status.code
    • A full list of tags can be searched within the input field.
  • Min Duration - The minimum duration of a transaction
  • Max Duration - The maximum duration of a transaction
  • Limit - How many results you wish to see

You can view the trace information by clicking on the trace ID, which will open a window on the right of the screen.

Log Queries

To explore logs, you can use the log browser.

The log browser allow you to filter and view any ingested logs.

Within the log browser you can:

  • Filter logs by any label, such as job, filename or instance.
  • Execute functions such as sum, count, or avg on logs to extract metrics from logs themselves
  • Combine multiple queries in a single view
  • View data as various chart types, or as raw data in a table.

The log browser will build log queries for you and automatically generate LogQL

You can also use explore to run your own LogQL queries not built using the browser.

LogQL is a query language developed by Grafana labs. It allows you to filter and search across your log content and can also generate metric data based on your log data.

LogQL examples

Below are some examples of LogQL.

Search for Logs with a specific job

{job="job1"}

Search for specific filename

{filename=~"request"}

Search for logs with a job using wildcard

{job=~"job.*"}

Search for any stdout or stderr logs for any jobs matching the wildcard

{job=~"job.*",filename=~"std.*"}

Query for the volume of jobs per job and instance

sum(count_over_time({job=~".+"}[5m])) by (job,instance)

Query for the top IP addresses triggering requests in FusionReactor

topk(10,sum by (clientAddress)(rate({filename="request"} | logfmt | line_format "{{.clientAddress}}" | __error__="" [1m])))

Query nginx for the top IP address hitting the load balancer

topk(10,sum by (remote_address)(rate({filename="/opt/access.log"} | logfmt | line_format "{{.remote_address}}" | __error__="" [1m])))

Search all logs for the text Exception { job=~".+"} |= "Exception"

Process the request log for top hit page URLs

sum by (url)(rate({filename="request"} | logfmt | line_format "{{.url}}" | __error__="" [10s]))

Graph the number of Error or Exception error lines

sum by (job) (count_over_time({job=~"store-.*"} |= "error" != "exception" [5m]))

Process the avg CPU time per page URL

topk(10, sum by (url)(avg_over_time({ filename="request"} | logfmt | __error__="" | unwrap cpuTime[5m])))

Graph previous crashes caused by OutOfMemory Errors

sum by (job) (count_over_time({job=~".+"} |= "java.lang.OutOfMemory" [1m]))