Skip to content

Alerting

FusionReactor Cloud Alerting allows you to create automated checks on the values of metric data gathered by a FusionReactor agent, and generate alerts when user-made conditions are met.

The alerting system is comprised of checks and subscriptions:

  • Checks are the conditions that are processed by the alerting engine.
  • Subscriptions can be added to a check to notify via other services when an alert is generated.

The alerting area can be found by clicking Alerting in the navigation bar, while alerting service configurations can be found by clicking Configurations.

It is required to set-up alerting services before you can start using them to make Subscriptions.

Checks

All checks on your account can be viewed, edited and deleted under the Checks tab (shown below), as well toggle their enabled state.

You may view a check in more detail by clicking its name on the left-side of the row.

!Screenshot

Creating a check

To create a check, click create check on the top right of page.

We recommend only creating checks that fire when action to correct a situation is required.

If you have too many alerts fire, we tend to find that alerts begin to be ignored. As a result crucial alerts that are important can be missed.

The alerting engine runs every 60 seconds.

!Screenshot

Step 1: Naming your check

The first step in creating a check is naming it.

Using a name that describes the situation firing the event is recommended.

Step 2: Building your query

The metrics list contains every metric that has been ingested into your cloud account

Functions can be applied to your metric if required, functions include:

  • Sum
  • Count
  • Average
  • Min
  • Max
  • Group
  • Standard Deviation

"Group by labels" will set which labels the data is grouped by, before a function is run on each group.

Any changes to your metric selection, function or group will be applied to the graph showing the metric state.

The timeframe of the metric can be adjusted from one hour, up to 2 weeks.

Filter with labels

The alerting UI will display each label associated with your selected metric.

You can select one or more label values to narrow down the data that's monitored by the alert. You can also add your own key/value pairs with the "Add a label key".

Selected labels will alter the metric graph values appropriately.

Set your thresholds

Using the metric graph for reference, you can set your alert threshold.

Use the selector to specify whether the value needs to be greater than, less than or equal to the threshold.

Add Subscriptions

Specify which of your existing subscriptions you would like your alert to be send to, or alternatively create a new subscription.

Optional configuration

You may set a summary and runbook for your alert notification. These are optional.

  • The summary is a short description of the check, so the receiver knows what the alert is about.
  • Runbook is for a URL. This is usually the location of the resources/instructions for "What to do when this alert fires".

The summary and runbook, if specified will appear in your alert.

Example Checks

When any instance goes offline for 5 minutes

The following example is a check that will monitor the online status of all FusionReactor instances using your Cloud license.

It is important to note that any instance which has been offline for over 5 minutes will stop being counted as offline by the check. To more reliably monitor the online status of a specific job/instance, see the check below It's also possible the graph will say "no data" when you are editing your check. This simply means no instances have gone offline within the timeframe shown.

!Screenshot

This check relies on the function "unless", which returns the value from 5 minutes ago when no data is present now. The Online (up) series is generated by FusionReactor, indicating that it is online. Therefore, if the instance was online 5 minutes ago, but isn't online now, it has gone offline and data will be returned. The condition is met so an alert will be fired.

When a single job goes offline for 5 minutes

This example will monitor the online status of an instance, and send an alert if it goes offline. If you choose more than one Job or Instance with these alert conditions, then all of the selected jobs/instances must be offline for an alert to fire. It's possible the graph will say "no data" when you are editing your check. This simply means the selected instance has not been offline within the timeframe shown.

!Screenshot

The important parts here are the function "absent", which returns 1 when no data is present, and the filter labels. The Online (up) series is generated by FusionReactor, indicating that it is online. No data will be present when the instance is offline, so the absent function will return a 1 and cause an alert to be fired.

When any instance is using over 90% CPU for 2 minutes

In this example, we make use of the threshold value to have an alert that fires if an instance is using over 90% of the system CPU.

!Screenshot

You can also use "less than" with the threshold for underflow alerts, to detect unusually low activity. This can be useful for services that often receive a lot of requests, as low activity could be a sign that requests are not reaching your service.

When any instance in a group is using over 90% allocation memory for 10 minutes

Instances can be assigned a group, this group will appear as a label. In the example shown, "testfr" is the group set on several instances, and each of those instances will be assessed by the check.

!Screenshot

Editing and duplicating checks

Editing a check will provide you with a form identical to the form for creating one. It will be populated with the existing configuration of the check and when you save it the original check is overwritten.

Duplicate similarly shows the same form with the check's settings already filled-in. Clicking save when duplicating will create a new check instead.

Alerting services

Subscriptions require a service they can send their subscribed alerts to. These are set-up in the configuration menu under Alert services

!Screenshot

The following alerting services are currently supported:

Configuring alerting services

Service configurations can be found under alert services in the Configuration menu.

From there you can configure the services you wish to use when creating subscriptions.

To configure a service simply click the Configuration button next to that service.

!Screenshot

A menu like the one shown above will appear. The required information differs for each service, and are explained in more detail in the sections below:

Email

  • Simply enable the service.
  • Email addresses are set during subscription creation.

Http (Webhook)

  • Enter Webhook URL in the subscription during creation.

Slack

More information: https://slack.com/

For the FusionReactor Cloud Slack app:

  • Click the 'Add to Slack' button which will redirect you to the Slack website.

  • If required, sign into Slack, then review the requested permissions and allow the app access to your workspace.

Note

The FusionReactor Cloud app will only have permission to view the name, email domain, and icon for your workspace. It cannot edit anything. For more information: https://api.slack.com/scopes/team:read

Note

If you have to request approval from your Slack workspace admin, you will have to follow these steps again. An admin can approve the request from the app management settings for your workspace or through a direct Slack message they will receive from Slackbot.

  • After granting the app access, you will be redirected back to FusionReactor Cloud and your service will be configured.

Disabling alerting services

Clicking the toggle next to the service will switch it between being enabled or disabled.

When a service configuration is disabled, the following will happen:

  • Each subscription that is linked to that service will be disabled, you will need to manually re-enable them after re-configuring the service.
    • You cannot re-enable the subscriptions until the service is enabled.
  • The settings saved on that service will be kept, so no extra set-up is required when re-enabling them.

Resetting alerting services

!Screenshot

You can reset the configuration for a service by clicking the Reset button next to that service.

When a service configuration is reset, the following will happen:

  • Each subscription that is linked to that service will be disabled, you will need to manually re-enable them after re-configuring the service.
    • You cannot re-enable the subscriptions until the service is configured and enabled.
  • The settings saved on that service will be deleted, so any API keys and other information will need to be entered again when re-configuring them.

Subscriptions

The subscriptions tab lets you manage and test your subscriptions.

!Screenshot

You may view a subscription in more detail by clicking its name on the left-side of the row.

Creating subscriptions

Clicking the subscription button as shown below, will open the form for creating subscriptions.

!Screenshot

Within this form you will be able to set the active times and days of the subscription, as well as which alert states it watches for.

!Screenshot

Messages will only be sent to the subscribed service of those conditions that are met by an alert. Each service has different properties that may be configured, and will be explained in more detail below.

Email

  • Subject is for the email subject, which is "FusionReactor alert" by default.
  • Addresses is where you list the email addresses of recipients for the subscription.
  • key and value fields let you enter internet/email message headers as key:value pairs.

Gitlab

Gitlab supports creating issues via email. Follow this guide to find out how.

Flowdock doesn't require any additional configuration beyond the service configuration.

HTTP Webhooks

  • Target URL is required, as it is the URL that the alerts will be sent to. Your endpoint should provide this URL that can be simply copied and pasted into the Target URL field.
  • Header is for the internet message headers attached to requests.
  • Body is for the body of the request.

The end-point of the webhook will vary on what it requires, so be sure to check the documentation of your endpoint to see what schema it follows. Then you can fill in the Body and Header to fit your needs.

Microsoft Teams

To set-up alerting for MS Teams, you will need to set up a Connector. Follow this guide to find out how. When done, it should provide the URL for your webhook.

Once you have the URL, you can enter it into your subscription as the Target URL, as mentioned above for webhooks.

Below is an example of the JSON schema used for MS Teams message cards. Copy and paste this into the Body of the webhook, or create your own. For more info on creating your own MessageCard follow this guide.

{
    "@type": "MessageCard",
    "@context": "https://schema.org/extensions",
    "summary": "FusionReactor Cloud Alert",
    "themeColor": "479eff",
    "sections": [
        {
            "activityTitle": "FusionReactor Cloud",
            "activitySubtitle" : "My alert title",
            "activityImage": "https://www.fusion-reactor.com/wp-content/uploads/2013/01/frico1.png",
            "text": "This text is here as a placeholder example of the webhook schema for MS Teams."
        }
    ],
    "potentialAction": [
        {
            "@type": "OpenUri",
            "name": "View alerts",
            "targets": [
                {
                    "os": "default",
                    "uri": "https://app.fusionreactor.io/alerting/alerts"
                }
            ]
        }
    ]
}

That looks like this:

!Screenshot

Slack

For the FusionReactor Cloud Slack app:

  • Click the 'Add to Slack' button which will redirect you to the Slack website.
  • If required, sign into Slack, and then select the channel you want alerts to appear in.
  • After, you will be redirected back to FusionReactor Cloud where you can continue to configure your subscription.

Editing and duplicating subscriptions

Clicking the edit button on the row of the subscription you wish to edit will present you with an interface just like the one used to create subscriptions. The values shown will match what's currently set on the subscription. Once you've made your desired changes, simply click save and your changes will overwrite on the subscription.

If you don't wish to overwrite the existing subscription you can click the duplicate button. The values shown will match what's currently set on the subscription you chose. You may make changes before saving it. When you click save at the end of the form, a new subscription will be made.

Deleting and disabling subscriptions

Clicking the delete button will cause a message to appear confirming that you wish to delete the subscription. Choosing Ok will delete the subscription from your account.

Warning

This cannot be undone.

Clicking the toggle under 'Enabled' will enable or disable the subscription. The check will appear blue when enabled. While disabled, checks will not send alerts to that subscription and the subscription will not appear when looking at a check in the detailed view. If the alert service used by the subscription is not configured, or disabled, then the subscription will be automatically disabled and cannot be enabled until the service is configured and enabled.