Cloud Status¶
The Cloud Status page (available for Cloud-licenses only) is accessed via the Cloud menu in the upper notification area, and selecting Status. The page shows you the current status of FusionReactor's cloud connections.
The page is split into a Health Check section and two subsections: one for the Control Channel and one for the Metric Transport.
Logging¶
There is a log available which records changes of cloud state. It's described here.
Health Check¶
The Health Check gives you a larger version of the cloud icon in the upper notification area, as well as some text to help you resolve any problems which may occur.
There are three failures detected by this section. We'll specify them in more detail later. They are:
- A failure of the Control Channel only - leading to a Degraded indication
- A failure of the Metric Transport only - leading to a Degraded indication
- A recovering Metric Transport, for example after a network outage, in which stored metrics are being transmitted – possibly out of order.
When the Control Channel and Metric Transport both fail, the status of the system changes from Degraded to Failed.
Control Channel¶
The Control Channel is used to provided a two-way communications channel between FusionReactor and FusionReactor Cloud. It's used to provide immediate, accurate information to the Cloud and is the basis for real-time functionality like Stack Trace, Running Requests etc.
This channel is established between FusionReactor and FusionReactor Cloud when the instance starts. FusionReactor tries to keep this channel open as long as the instance is running, and will attempt to reconnect it if it becomes unavailable.
Displayed Data¶
Not all fields are always displayed. See the Appears column for when you can expect to see that item.
| Text | Appears | Description |
|---|---|---|
| Status | Always | A short description about the state of the connection. |
| Endpoint | Always | A URL describing the server, protocol and port used by this instance to establish a connection to FusionReactor Cloud |
| Connected Since | Connected | A date/time string representing the instant at which the connection was established. |
| Disconnected Since | Disconnected | A date/time string representing the instant at which the connection was lost. |
| Attempting to Reconnect | Disconnected | Yes/No. Indicates whether the Control Channel system is attempting to reconnect. This should normally be Yes. Some types of forced disconnect may result in No, however. If this is the case, FusionReactor will attempt to reestablish the connection only on the next instance restart. |
| Last Exception | Always if an Exception has been caught | If the Control Channel system has encountered any exceptions during operation, the last one is displayed here, along with its causes (if any). |
Troubleshooting¶
| Failure Mode | Presentation | Meaning and Resolution |
|---|---|---|
| Network - DNS | An exception of the form "Can't resolve xx.yy" | The endpoint couldn't be resolved from a name into an IP address. Ensure the instance is configured to use a DNS server that can resolve standard IP4 internet addresses. Ensure the instance is connected to the network. |
| Network - Connection | An exception of the form "Couldn't connect to xx.yy" or "Timed out connecting to xx.yy" | The endpoint couldn't be contacted. Ensure the instance is connected to the network. Ensure the instance can connect through your firewall to the Control Channel. Information on required firewall punches and static routing (if required) can be found in technote [FRS-418] FusionReactor Cloud Firewall DNS and Static IP address rules. |
| FusionReactor Cloud may be experience network connectivity issues. In this case, the connection will be periodically retried by FusionReactor, and will reconnect up to a maximum of one minute after FusionReactor Cloud becomes available. |
Metric Transport¶
The Metric Transport system periodically collects useful statistics from FusionReactor's metrics-gathering system, packages them into datapacks and sends them to the Cloud. This happens once per minute. FusionReactor Cloud them ingests this data and makes it available to you in various useful forms.
Metric Transport is reliable in that any datapacks that can't be transferred are stored in a queue. The queue is a moving 60-minute window. This means that in the event of a failure, 60 datapacks are automatically stored. If the failure continues for longer than that, earlier datapacks are dropped to make room for newer ones.
When the connection returns, Metric Transport sends any pending messages in reverse order: newest first. During the period of outage, you may see holes appearing in metric data within FusionReactor Cloud. Once these messages start flowing again, these void areas should close, filling gradually backwards from the current time. While this backfill occurs, you may see a Metric Lag Warning on this page.
The Metric Transport connection is not permanent but is established every minute, and is only alive for as long as it takes to transport the datapack. For this reason, the health of the connection is effectively unknown for the first minute the instance is alive. Only when the first transmission is attempted can the status of the connection be definitively known.
Displayed Data¶
Not all fields are always displayed. See the Appears column for when you can expect to see that item.
| Text | Appears | Description |
|---|---|---|
| Status | Always | A short description about the state of the connection. Unlike the Control Channel, the connection begins in state Not Yet Attempted until the first datapack is transmitted. |
| Endpoints | Always | A list of ingest endpoints used by the transport. The system supports using multiple transports (although currently only one is used), so you may see more than one item listed here. |
| Pending Messages | Always | The number of datapack messages currently in the queue. During normal operation, this should be 0. |
| Last Successful Transfer | Always if at least one datapack has transferred | A string version of the Transfer Result generated by the system. DTS: The timestamp of the datapack contained in this transfer. State: A short String describing the state of the transfer. By: The name of the Transport that actually moved the datapack. |
| Last Failed Transfer | Always if at least one datapack has failed to transfer | A string version of the Transfer Result - see above. |
| ... Transfer Receipt | Always if at least one datapack has failed to transfer | A string representation of the transport-dependent receipt for the transfer. This usually contains information about why the transfer failed. |
| Rate Limiting | Always | If FusionReactor Cloud is experiencing high load, it may limit transfers from one or more clients. If the current instance has been rate limited, the date/time of the next allowed transport is listed here. FusionReactor will pause transfer attempts until this time arrives. |
| Last Exception | Always if an Exception has been caught | If the Metric Transport system has encountered any exceptions during operation, the last one is displayed here, along with its causes (if any). |
Troubleshooting¶
The failure modes for this transport are similar to the Control Channel above. Again, FRS-418 may be helpful if your firewall is not allowing outbound connections from FusionReactor.
Support Functions¶
There are two support functions available at the top of the page. Automatic Refresh can be set using the dropdown at the top right, and Copy to Clipboard will copy all displayed information into the clipboard, in a format our support team can use to help you diagnose issues.