SENSE RTMon

SENSE RTMon automates real-time monitoring and visualization of the network services provisioned by the SENSE Orchestrator. It polls the orchestrator for tasks assigned to it, retrieves the manifest of each corresponding service instance, and builds one Grafana dashboard per instance covering the whole provisioned path rather than one site at a time.

How It Works

Panels are drawn from three sources: the per-site Prometheus exporters (SNMPMon for switches, node exporters for hosts), the SiteRM frontend of each site on the path, and the instance manifest itself. RTMon does not measure anything of its own; it locates what each domain already publishes and assembles it against the path.

Dashboards are written to a single Grafana folder, named Real Time Mon unless the deployment overrides it. A dashboard is created when monitoring is enabled on an instance and removed when the instance is cancelled, unless retention was requested.

For the operator-facing configuration of the addon, see Addon Configuration.

Enabling Monitoring for an Instance

RTMon is offered as an addon on the orchestrator. Monitoring is enabled per service instance, and the options below are submitted with it. They can also be set in a service profile so that every instance created from it carries the same choices.

Option	Effect
Ping Test from Hosts	Pings between the endpoint hosts of the path, repeated every 5 minutes for as long as it stays enabled.
Traceroute from Hosts	Traceroute between the endpoint hosts, repeated every 5 minutes for as long as it stays enabled.
Ping Test from Network	Pings from the network endpoints. Runs once, for 5 minutes. To run it again, disable and enable it.
Traceroute from Network	Traceroute from the network endpoints. Runs once. To run it again, disable and enable it.
Performance Test	A throughput test between two endpoints, using iperf3, FDT, or Ethr. Streams defaults to 4 and is capped at 16, runtime defaults to 600 seconds and is capped at 1800, and the test may be run in both directions at once.
Dashboard Retention	How many days to keep the dashboard after the instance is cancelled. 0 removes it with the instance. The deployment sets both the default and the ceiling, and a request above the ceiling is reduced to it.
Show All Learned MACs	Adds a table of every MAC address the devices on the path have learned, not only the ones belonging to this instance.
Debug Mode	Renders the detailed dashboard: the additional topology diagram, the L2 and L3 debugging detail, and the manifest and instance output as text.

The results of the tests above are charted on the dashboard itself, in the Debug Action Results row, so a test does not have to be correlated against a separate tool.

Note

The tests are submitted by RTMon directly to the SiteRM of each site, which a deployment can hold off. When it does, the orchestrator shows the option with (temporarily off) appended and a description saying so. Selecting it in that state has no effect, and nothing is submitted to any site. The options that only shape the dashboard, which are retention, Show All Learned MACs, and Debug Mode, are never affected by this.

Dashboard Overview

Rows appear in this order. Hosts and switches are numbered along the path, so the row titles follow the direction of the circuit rather than an alphabetical list.

Row	Contents
Topology diagram	The path as a Mermaid diagram, built from the manifest.
N. Host Flow Summary: <site> <host>	Interface counters and errors for one endpoint host, plus the annotations marking when a test ran.
N. Switch Flow Summary: <site> <host>	Interface counters for one switch, narrowed to the ports and VLANs this instance uses, alongside the QoS reservation reported for that VLAN.
N. Switch BGP Summary: <site> <host>	Peer session state, accepted and advertised prefix counts, and the age of the last scan, for switches whose site exports BGP session data.
L2 Debugging:	The MAC addresses being searched for, listed at the top of the row, and then where each was learned along the path.
Debug Action Results	The output of the tests this instance enabled, once the first run lands. An empty table means no run has returned yet.
SiteRM Service Warnings	What each site on the path says about its own services. Only services that are not OK are listed, so an empty table means every site reported OK.
All MAC Addresses (<site>, <host>)	Present only when Show All Learned MACs is enabled.
Debug Info (SENSE-O Output)	The manifest and instance as returned by the orchestrator. Present only in Debug Mode.
Graph Generation Warnings (N)	Every panel that was left out of this dashboard, and why.

A row is omitted when the path has nothing to put in it. A path with no BGP anywhere on it has no BGP row, and a site that exports no QoS reservation for the VLAN contributes no QoS panel.

Missing Panels

A panel with no monitoring behind it is not drawn. This is deliberate: a flat line at zero reads as a dead transfer, and missing monitoring should not be indistinguishable from a circuit that is carrying nothing.

Whatever is left out is recorded in the Graph Generation Warnings row at the bottom of the dashboard, with the reason and the device or site it concerns. That row is the first place to look when an expected panel is not there.

Only a definite answer hides a panel. If a site’s exporter cannot be reached at all, nothing is hidden, because “this device reports no such metric” and “nobody could ask” are different answers and are not treated alike.

Lifecycle of a Dashboard

• The dashboard appears shortly after monitoring is enabled, and its URL is reported back to the orchestrator on the task.
• It is regenerated in place as the deployment’s dashboard template advances, so panels added in a later release reach dashboards that already exist. The dashboard keeps its place in the folder while this happens.
• On cancellation the dashboard is removed, unless a retention period was requested, in which case it stays under the same name and is removed when the period expires. The history of a finished transfer therefore outlives the instance only if it was asked for in advance, before the instance was cancelled.

Troubleshooting

Symptom	What it means
A panel or a whole row you expected is not on the dashboard	Look at the Graph Generation Warnings row, which names what was left out and why. A site that does not export the metric is the common case.
A test option is labelled (temporarily off)	The deployment is holding the SiteRM tests off. Enabling it has no effect until the operator turns them back on.
A test was enabled, but Debug Action Results is empty	No run has returned yet. The network tests run once, and the host tests are repeated every 5 minutes, so the first result is not immediate.
The dashboard disappeared when the instance was cancelled	No retention was requested. Retention has to be set before cancellation, since it is read from the instance’s settings at teardown.
Monitoring was enabled, but no dashboard was ever created	This is an orchestrator-side or deployment-side mismatch rather than anything selectable here. See Addon Configuration.

Related Documentation

• Addon Configuration for the orchestrator-side settings.
• Monitoring and Health for the orchestrator’s own monitoring.