SENSE Operations Manual | Release Notes

Release 26.09

Tue, 29 Sep 2026 00:00:00 GMT

Technical Release Details: https://github.com/esnet/StackV/releases/tag/26.09

Release 26.09 is our Q3 quarterly release, introducing integrated Project tracking, per-task access controls, and a large expansion of IRI and WAN Data Fabric orchestration. And as this release is part of our year-of-production sprint, we naturally also have major improvements to orchestrator model stability and observability.

⚠️ Breaking Changes

Task endpoints now enforce individual task access. Reading a task requires READ access, updating its status or message requires WRITE access, and archiving requires OWNER access. API clients that previously relied on broad access to arbitrary tasks may now receive 401, 403, or 404 responses.
- To remedy this, you may need to adjust the task’s tag or establish access control policies as needed.

✨ New Features

Projects are now a first-class concept in SENSE-O. Administrators can record project definitions and associate them with users and groups, which will then be associated with any licenses or instances they create. A new project association mode controls how those associations are applied along a new set of /project API endpoints.

Task Access Controls bring the same ownership model used elsewhere in SENSE-O to Tasks. Owners can delegate read, write, or owner access to other users directly from the task UI, and task rows now include a quick link to navigate to their parent.

IRI and WAN Data Fabric orchestration has been significantly expanded:
- L3VPN support for the WAN Data Fabric, including optional BGP MD5 authentication keys, also available as an IRI Facility Space network parameter.
- L3 routing over facility interconnects, with local and remote addressing defined from the facility’s perspective.
- A new facility endpoint type for WAN Data Fabric connections, and improved subnet resolution for L2VPN and job placement.
- New IRI facility routing and compute switching services.
- Support for the IRI v2 API, with continued support for v1.

Observability improvements across the portal and backend:
- Instances now have clearer observability and audit tracing in the details view.
- System Health now reports more comprehensive monitoring information, along with diagnostic model metrics.
- The Driver table now shows overall health metrics across the board.
- Model-cache resets are faster, and the cache layer is more resilient under unstable conditions.

Minor Additions

The orchestrator container now ships with a built-in healthcheck against the version endpoint, and our Docker Compose templates have been updated to match. Operators maintaining their own Compose or Helm definitions should adopt the new check.
The JVM now writes a heap dump and exits on an OutOfMemoryError instead of stalling, keeps a rotating GC log in the server log directory, and warns at startup if JAVA_MEMORY is set below the recommended floor.
Container images now include SBOMs and better multi-platform support.

🐛 Bug Fixes

Fixed HTTP client starvation when dozens of authentication challenges ran simultaneously.
Fixed address pools allocating subnets outside their configured range.
Fixed several model-cache miss and lock issues that could stall orchestration.
Fixed slow portal loading and redundant requests when Keycloak is under load.
Fixed a client leak in the user service.
Fixed a stale layout state that could break the visualization when nodes were missing.
Fixed soft-capped QoS not always releasing bandwidth.
Timekeeping and date rendering are now centralized across SENSE-P for consistent display.

🚧 Known Issues

Project synchronization from external sources is not yet available, but is being actively developed.

Release 26.08

Thu, 03 Sep 2026 00:00:00 GMT

Technical Release Details: https://github.com/esnet/StackV/releases/tag/26.08

Release 26.08 is a minor release focused primarily on our long-overdue migration to MySQL 9.7, including improved system stability, health visibility, a healthy mix of UX and UI improvements.

⚠️ Breaking Changes

Due to incompatibilities and complications with our schemas, a fresh installation is required for this release. There is no supported in-place database upgrade from existing deployments.
- Use the new System Backup and Restore feature and the legacy archive exporter to transfer some of your existing data to the fresh installation.
  - Archives include selected system settings, drivers, and service profiles.
  - Archives exclude service instances, tasks, alerts, model/runtime state, metadata, logging, and audit history.
The SENSE-O Docker Compose templates and Helm charts now use MySQL 9.7, and Flyway has replaced Sqitch for our DB migration sidecar.

✨ New Features

A new System Backup and Restore feature has been added, allowing operators to transfer data between SENSE-O deployments or to save a snapshot of their current configuration. For more details see the dedicated page at System Backup and Restore.

System Health telemetry for Version Groups, Version Items, and model rows has been added. This allows managers to inspect and monitor key model statistics, as well as inspect the new model-cache state.

The new Profile Table page has replaced the previous pop-up modal, and further support for navigation, saving and tagging features, and general usability improvements have been made.

As mentioned above, the SENSE-O application stack now operates on the latest LTS of MySQL. This naturally brings improved database performance, but a large amount of work was done to optimize and refactor many parts of our persistence code that were lacking or problematic. The result is a more consistent and reliable orchestrator, and a large chunk of years-old tech debt finally overcome.

Minor Additions

A pilot Garbage Collection system has been added, allowing for scheduled cleanup of eligible orphaned and stale model/version artifacts. Currently, this system is automatically operated in the background, but API endpoints allow admins to manually execute cleanups as needed. In the future this system will be integrated into a more robust background tasks system, allowing operators to configure and monitor the runs of various system operations.
The SENSE-O RBAC has been updated to support more restrictive and explicit user permissions, based around user classes. This is the first step in a larger user management rework we are undergoing, with more significant improvements to come later this year.
Standalone SENSE-P developments can now select a remote orchestrator to connect to, even across Keycloak boundaries.

🚧 Known Issues

The orchestrator may still run into VersionItem or VersionGroup stalls in rare circumstances. This release adds more measures to allow for remedies without restarting the orchestrator (such as the new system health and garbage collection), as well as better transactional stability. Still, an operator may run into cases where service instances are unable to be provisioned (often accompanied by orchestration errors like VG_VI_ERROR). In such a case, if the unplugging and re-plugging of underlying drivers does not resolve the issue, a restart of the orchestrator may be required.

Release 26.06

Tue, 30 Jun 2026 00:00:00 GMT

Technical Release Details: https://github.com/esnet/StackV/releases/tag/26.06

Release 26.06 is our Summer release for CY2026, and it continues the major work we started earlier in the year to modernize the SENSE portal and access control. With this release we’re also bringing in our first work items built to support AmSC and IRI workflows.

This release keeps the focus on steady platform improvements. The biggest changes are in the areas of UI consistency, backend workflow handling, our new access control system, with our normal number of smaller tweaks and fixes folded in to reduce UX friction.

🚧 Known Issues

Depending on your browser and reload frequency, you may sometimes encounter a visual error when deleting service instances from the instance table. If this occurs, reload the page to resume operations.
Due to the transition to our new Keycloak server version, some users may find they are missing permissions they previously had. If this occurs, please contact your system administrator for assistance.

✨ New Features

A brand new RBAC system that allows for more granular control over user permissions and increased security via segregation of user classes. This is mainly a transparent feature for the average user but brings with it better support for more complex workflows and more structured policy controls.

A reconfiguration and cleanup of our logging backend, which should improve performance and allow system admins to more effectively monitor and troubleshoot system issues. This update allows admins to dynamically set logging levels at runtime for individual classes and packages. No more needing to turn on trace logging across the entire orchestrator to find that one driver-level introspection.

A new pilot global audit record stack has been implemented, allowing users to track what events happen with their instances and allowing admins to trace system-wide operations and modifications. This system can be accessed via the “View Operations History” button in the service settings, or via the new “Operations History” sidebar link.
Currently, the operations being tracked are limited to basic service instance operations, but more operations and more domains (such as Profile or Driver operations) are planned to be added in future releases later this year.

The first pilot of our project support has been implemented, allowing us to associate service instances with external projects. Intents can now carry a project ID that will be injected into service metadata, and NSI drivers can specify a default project ID to use when this intent field is empty. This functionality is generic, and will be expanded to other applicable drivers. In the future we will have a more robust selection UX, so that users can pick from a list of more recognizable names and state on their own which project their service is associated with.
Currently, the system is mainly manual, but we plan to explore options to automatically fetch or retrieve project IDs and metadata from places like ESDB.

Minor Additions

A Download CLI Config button has been added to the navbar to help users quickly template a suitable CLI configuration, for use with the SENSE python client.
The Task subsystem has been expanded with support for job dispatching, and improved metadata controls.
The Visualization subsystem has been tweaked to improve performance, and a new line rendering system has been added to help with consistency on larger networks and topologies.

🐛 Bug Fixes

Fixed address pool allocations not properly displaying details such as affiliates.
Fixed address pool allocation logic and reconciliation during expiration situations, and when affiliate data is not present.
Fixed some gaps in NSI driver logging.
Fixed an error with Fabric driver modify operations.
Better associate driver metadata with relevant service instances, and vice versa.

Release 26.03

Sat, 28 Mar 2026 00:00:00 GMT

Technical Release Details: https://github.com/esnet/StackV/releases/tag/26.03

Release 26.03 is our regular Spring release for CY2026, and includes a number of new features and bug fixes to support ongoing efforts to modernize the SENSE portal and support our growing external ecosystem.

🚧 Known Issues

Due to some environmental factors, the retrieval of users and groups by some SENSE portals can be slow or lag somewhat.
- This will be remedied in the next release, and in the meantime you may need to allow 10–15 seconds in some cases for the user or group lists to connect and populate.
Some parts of the SENSE portal user interface may seem out of place as we transition to a new UI framework and modern style, such as the current process for managing profiles. These components will be slowly improved and brought into alignment with the rest of the portal over the course of the year.
Though less frequent, the intermittent problems with the SENSE modeling may still be present and require a reload or refresh of the orchestrator.
- Resolution can also be attempted by plugging or unplugging drivers to force a new global model creation, or by creating or terminating instances.

✨ New Features

We have added a brand new underlying user interface framework designed to be more intuitive and user-friendly. This new system will be expanded over the rest of the year to slowly replace the existing UI, with our main surfaces and tables already using this new system.

A new dedicated SiteRM driver has been added to support the new SiteRM API authorization scheme and any other future improvements it may bring.
- This new driver can operate alongside the legacy SENSE-RM driver, which is now considered deprecated.

Our task distribution service has been expanded with more complex and resilient workflows to support our new external demands and use cases. We will be considering future expansion of this subsystem as needed and expect further demand as other projects advance and evolve.

🐛 Bug Fixes

Fixed various bugs with the visualization system rendering pipeline that were causing performance issues.
Added missing traceability and principal parameter chaining to the driver level, so that we can allow drivers to access the requesting principal.
Fixed permissions issues causing edge case blockages, specifically during instance operations.

Release 26.01 (Final)

Wed, 21 Jan 2026 00:00:00 GMT

Technical Release Details: https://github.com/esnet/StackV/releases/tag/26.01

Release 26.01 is a polishing follow-up to our previous alpha release, 25.12. The main improvements for this release are related to system and model stability, but there are a number of usability and interface changes present as well.

🚧 Known Issues

The observed performance issues during some high-concurrency operations remains under investigation, though they should be improved from the previous release. Avoiding large-batch service provisioning or spacing out requests should alleviate any related problems.

✨ New Features

(Beta) A new health monitor has been added in the prototype stage, for now focused on piloting how to internally trace and track health metrics such as dynamic entity counts or stage timings.

🐛 Bug Fixes

Major improvements to persistence performance and stability, specifically focused on the VersionGroup and VersionItem entities.
Cleaned up various NPEs in async sections of orchestration.
Fixed performance losses and major glitching of visualization rendering.
Improved behavior and UX of visualization features introduced in 25.12.

Release 25.12 (Alpha)

Tue, 23 Dec 2025 00:00:00 GMT

Technical Release Details: https://github.com/esnet/StackV/releases/tag/25.12

The SENSE-O 2025 Q4 Release is out now! This is our first release using our new versioning and documentation pattern. Below you will find highlights of the new features, resolved bugs, and upcoming enhancements. We will continue to refine and expand the structure and style of these notes as we move forward, including more detailed change notes, example images for UX changes, and more.

The main highlight for this release is a somewhat subtle one, as we committed to a long-overdue upgrade of our underlying development and language foundation, updating our Java, JPA, and WildFly dependencies to modern LTS versions, in some cases jumping over ten years of work. This was a tenuous affair, and while it has provided us with a large amount of breathing room in terms of future development and maintenance of our code base, the wide-reaching nature of the changes have warranted us spending a couple extra weeks refining and fully testing the results.

As with all of our recent releases, this release also includes numerous additions for new pilot subsystems, as well as expansions of existing addon elements to support our ever-growing catalog of use cases and our upcoming work to support our third SENSE Addon.

⚠️ Breaking Changes

The orchestrator is now built on Java 21, and utilizes the Jakarta EE and Hibernate 6 development stack. This change is a development-level change and only affects development and direct-install environments.

🚧 Known Issues

Occasionally the orchestrator’s backend model can end up in a stalled or locked state, due to a pervasive issue within our JPA persistence layer that has been unveiled by this release’s framework update. This will be addressed by the final release next month.
- Resolving this requires allowing the orchestrator to create a new model version object, either by unplugging and resetting the server drivers, or by instantiating and cancelling a functioning service instance or service profile in order to force a new change.
There are some intermittent performance issues associated with high-concurrency service requests (in excess of 5 or more instances). These are assumed to be side effects of the above JPA hiccups and persistence-level inefficiencies.

✨ New Features

The project is now running on a slew of updated underlying dependencies, such as Java 21, Hibernate 6, Jakarta EE 10, JAX-RS 2.
The Janus and RTMon Addons now support intent-level configuration, allowing for auto-registration or other options to be defined at service creation or within a service profile.
Permissions and ACL rights have been refactored to focus on User Groups as a primary principal. Users can now belong to groups and inherit permissions from them, and various system elements can now be shared with groups as well as users. This is the first step towards a future feature to have more complex policy controls.
The portal’s Visualization component has received multiple overdue tweaks and improvements, including a new subview system that allows user to arrange individual domains or nodes to their liking within a limited window, as well as layout changes and greater usability for search and node interactions.
The portal now provides a simple UI for Admins to allow access to the audit logging for instances and drivers. This will be replaced with a fully-interactive table in the future once the common work has been completed.

🐛 Bug Fixes

Fixed some flaws in the bandwidth calculation algorithm for path capacity and over-subscription cases.
Fixed an edge-case that allowed delegated users to edit service profiles via the use of custom editable fields.
Fixed a small bug with AL2S drivers that would incorrectly block valid multi-point services.