Releases
Observability Module for Ignition 8.3
Every published version of the module, with what it has been verified against, plus direct downloads for the .modl asset, the user manual and the Grafana dashboard pack.
v0.10.0
v0.10.010 Nov 2026OpenTelemetry and Prometheus observability for Ignition 8.3: curated gateway metrics, logs, traces and opt-in audit export, pushed to any OTLP backend or scraped on :9464.
- Ignition:
- 8.3.0+
- Backend:
- Any OpenTelemetry HTTP/protobuf endpoint, or Prometheus scraping the gateway
- Other:
- Ignition Edge: contact Mustry Solutions before deploying
Changes
- The license text changed (section 1: contact Mustry Solutions before installing on an Ignition Edge gateway), so the gateway asks you to accept the license again on upgrade.
- Breaking: three gateway network counts are now counters, so on the Prometheus endpoint they gain _total: ignition_gateway_network_acks_expired_total, ignition_gateway_network_messages_expired_total and ignition_gateway_network_messages_non_protobuf_total. The old names are no longer served. Update dashboards, recording rules and alerts that use them, preferably to rate() or increase() of the new names. The bundled dashboards and alert rules do not use them, and the OpenTelemetry names do not change.
- Breaking, for scripts that call the settings API (POST /data/mustry-observability/settings): an unknown key, a value of the wrong type or out of range, overlong text, or a secret object without a mode now gets HTTP 400 naming the field. These used to answer 200 and were ignored or stored as something else. Send {"mode": "none"} to clear a secret and leave a field out to keep it. The Settings page is not affected.
- Breaking: Save now refuses an Auth Header Name that HTTP itself uses (such as Host or Proxy-Authorization), an OTLP endpoint with a user name, password, query or fragment in it, and a secret with a line break, a control character or a non-ASCII character. A value like that stored by an earlier version can stop that destination, with an error in the gateway log saying what to correct, so check the Export Status page after upgrading.
- Breaking, for script metrics: at most 500 script metric names are exported (-Dmustry.observability.scripting.maxMetrics sets another limit), and a gauge keeps its 1,000 most recent label sets. When two names are served as one, because they differ only in letter case or separators, only the first one written is exported. Refused names are listed in /health (scriptMetricConflicts) and on the Export Status page. Keep IDs out of metric names and label values.
- Import the Grafana dashboards and copy the alert rules again. The dashboards now have Prometheus and Loki datasource selectors, so they work in any Grafana, and Gateway Overview has a gateway selector. A certificate renewal that changes the issuer no longer fires IgnitionCertificateSeriesVanished.
- When the Prometheus port is already in use, everything else keeps running. The module tries the port again until it is free, and the new health tag Prometheus/FellBack shows it. Before, the whole module stopped.
- Saves are recorded: each Save stores who made it, and a Save that changes a secret logs who changed it, never the value.
- Log events the module failed to export are now counted in /health (logsDroppedByError) and in the new health tag Export/LogsDroppedByError. The log drop counts now run from the module start, so a Settings save no longer resets them.
- Fixed: Test Connection could be made to read any secret in any secret provider, a database password for example, and send it to any host. It now uses a referenced secret only when it is the one saved for that field.
- Fixed: an OTLP or Azure Monitor endpoint that answered with a redirect lost every batch while export read healthy. The module no longer follows redirects; Export Status says where the redirect points and what to enter instead.
- Fixed: an empty -Dmustry.observability.prometheus.token served /metrics without authentication. The endpoint now stays off with an error. Invalid port and log level properties are reported instead of misread.
- Fixed: a Prometheus scrape that asked for protobuf first, as some Grafana Alloy and Mimir setups do, got HTTP 500. A scraper that gave up mid-answer no longer writes a stack trace to the gateway log.
- Fixed: two script metric names served as one Prometheus name made every scrape fail with HTTP 500 until a restart, and two spellings of one label key took the metric off the endpoint. Script metrics with NaN values, number or boolean labels, or label keys without letters or digits are now handled and reported.
- Fixed: a gateway clock set back, such as by an NTP correction, made the module drop almost every log event under a Rate Limit, lose audit records, keep sending an Azure token that had expired, and show negative health ages.
- Fixed: a Settings save, license change or redundancy change lost the audit records written since the last poll, could make the health tags read healthy while export was failing, and could leave the module on older settings. Two settings API requests at the same time no longer lose a change.
- Fixed: log events without a message, such as log.error(e.getMessage(), e) for most exceptions, were never exported. During a burst of logging from many threads, those threads no longer wait for each other in the module's log export.
- Fixed: ignition.alarms.events counted alarms twice in a fleet view, through a redundant backup or a hub with remote tag providers.
- Fixed: the MQTT Transmission message and byte counters could go down and show false spikes, and an edge node whose online state could not be read counted as offline.
- Fixed: with Azure Monitor direct, a Settings save during a Microsoft Entra outage threw away a token Azure still accepted, and the reason a token could not be fetched now shows on Export Status.
- Fixed: on a gateway with no devices, occasional failed polls could disable the device collector days later.
Known issues
- On Ignition versions before 8.3.6, some curated metrics are missing because the gateway does not provide them (16 on 8.3.0, down to 9 on 8.3.4). Up to 8.3.3 this includes faulted database connections, so that alert stays silent. Use 8.3.6 or later for full coverage.
- The certificate metric reads the gateway's HTTPS listener on localhost. A gateway whose HTTPS port is bound to one specific network interface reports no certificate series.
- Anyone allowed to use Test Connection can send the saved secrets to a host they name. Give the module's write permission only to people you would trust with those secrets (user manual §3.1).
- The Settings and Export Status pages use undocumented parts of the gateway web interface, tested on 8.3.6. A later 8.3.x update could break those pages. Export itself would keep working.
- Traces are manual only, through system.observability.span(). Ignition offers no way to trace HTTP requests, database queries or scripts automatically.
- Spans sent straight to Datadog get no APM trace metrics.
- Azure Monitor's OTLP ingestion is still a Microsoft preview.
- Proxy authentication and client certificates (mutual TLS) are not supported yet.
- The curated metrics do not yet cover SQL Bridge, EAM, Sepasoft or the remaining Cirrus modules.
v0.9.0
v0.9.023 Oct 2026OpenTelemetry and Prometheus observability for Ignition 8.3: curated gateway metrics, logs, traces and opt-in audit export, pushed to any OTLP backend or scraped on :9464.
- Ignition:
- 8.3.0+
- Backend:
- Any OpenTelemetry HTTP/protobuf endpoint, or Prometheus scraping the gateway
- Other:
- Ignition Edge: contact Mustry Solutions before deploying
Changes
- Breaking: the gateway CPU metric (ignition.gateway.cpu.utilization, ignition_gateway_cpu_utilization_ratio in Prometheus) is now a 0 to 1 ratio, as documented. It used to be 100 times too high. If you alert on it or chart it in your own dashboards, divide your thresholds by 100 (> 80 becomes > 0.8). The bundled dashboards now show it correctly.
- Breaking: the project, historian, provider, engine and queue labels now keep the name as you configured it. They used to be lower-cased with punctuation turned into _, so two instances such as Line-1 and Line_1 could share one series and lose data. A query that filters on a value needs the new spelling, for example provider="System" or queue="Tag - default - DB Values". To cover the upgrade, match both: provider=~"system|System". The bundled dashboards and alert rules need no change.
- Breaking, for redundant pairs that push over OTLP: the backup now reports itself as <master name>/backup instead of using the master's name. Before, the data of both nodes mixed into one series, and the backup-active alert fired on a healthy pair. Scraped Prometheus data is not affected.
- Breaking, for custom metrics on the Prometheus endpoint: names and label keys with repeated punctuation convert differently (line..1 becomes line_1, and a label key that starts with a digit gets key_ in front). Two custom metrics that now convert to the same name make the whole /metrics scrape fail until one is renamed. Use letters and digits with single ., _, - or / between them. The curated metrics and OTLP push are not affected.
- Breaking, if you upgrade from 0.7.0: export health is now tracked per destination. Export/MetricsHealthy and Export/LogsHealthy are true only when every enabled destination is healthy, and a destination that cannot read its secret switches off on its own while the others keep running (alarm on the new FailedClosed tags). Gateways with one destination see no change.
- If you upgrade from 0.7.0 and your backend copies resource attributes onto series (the Grafana OTel-LGTM stack does), every series starts fresh once, because telemetry now also carries service.version, host.name and the module version. Graphs recover on their own within minutes. Plain Prometheus scraping is not affected.
- If you copied the bundled alert rules into your own setup, copy them again. IgnitionDeviceFaulted now also fires for a disconnected device, the most common failure. IgnitionClockDrift could never fire before and now counts drift events. The certificate rules now filter on scope.
- The /health endpoint reports pushEnabled instead of otlpEnabled. Update any script that reads it.
- Data minimisation, all off by default: pseudonymise user names (the same user always gets the same pseudonym, so counting still works), redact IP addresses, limit which MDC keys are exported, and drop stack traces. One setting, Data Profile: Minimal, turns all of them on at once, and a settings lock stops anyone loosening them from the Settings page. User manual §3.11 and §3.12.
- "What leaves this gateway": the Export Status page now lists what the module actually sent in the last hour, per signal and destination: hosts, metric names, attribute keys and series counts, never the values. Download it as JSON for a security or supplier review.
- New metrics, with example alert rules in the Grafana pack: latency percentiles for database queries, scripts, the historian and Perspective; load per database connection; JVM memory, garbage collection and class loading; redundancy role and state, and the status of each gateway network connection; licence mode, edition and installed modules; gateway health-check problems; scheduled backup age; and an alarm activation counter for alarm-flood alerts.
- Certificate expiry: the gateway's HTTPS certificate, its gateway network certificate and its trusted peer certificates, with alert rules for certificates that are about to expire.
- Connection events: with log export on, every database, device or OPC connection that changes state sends one log event naming the connection, so an alert can say which one faulted.
- Sending data through a corporate network: new settings for an HTTP proxy (without authentication) and for extra CA certificates, for proxies that inspect TLS. No more editing the gateway's Java trust store.
- Pushing straight to Datadog now works, and direct push to Grafana Cloud, Datadog and New Relic is now tested against live accounts.
- Exports ride out short outages: a refused export is retried for about 15 seconds. For longer outages, the user manual (§4.10) shows how to put an OpenTelemetry Collector with a disk-backed queue in front.
- Generic OTLP export is now gzip-compressed, so less data leaves the site.
- Fixed: health tags no longer read as failing for 15 to 25 seconds after every gateway restart or settings save, so alarms on them no longer go off each time.
- Fixed: a destination that failed to start, or log export that stopped because its pseudonymisation key was missing, looked like a part that was switched off, so no alarm fired. New health tags Export/OTLP/FellBack, Export/Azure/FellBack and Export/LogsFailedClosed show it.
- Fixed: Azure Monitor no longer receives the full total of every counter as one spike after a settings save or restart, which set off false alerts.
- Fixed: a disabled OPC connection counted as faulted, and restarting the gateway while a database was down raised the fault alert a second time.
- Fixed: a stored setting that the new save checks would refuse made the gateway fall back to defaults after a restart. That dropped the Prometheus bearer token and served /metrics without authentication. The stored settings now load, and only the affected destination stays off.
- Settings saves are now checked, with an error under each field that is wrong. Pressing Save with nothing edited applies a secret rotated in its provider. On a redundant backup, a change is refused with a clear message to save on the master.
- Test Connection now says what is wrong: a host name that does not resolve, an unreachable proxy, or an untrusted certificate, instead of a generic failure.
- Fixed: an https:// endpoint named without a dot, such as a Docker or Kubernetes service name, failed to export while Test Connection said it would work.
- Fixed: a failed Microsoft Entra token refresh threw away a token that was still valid, so Azure rejected data until the next successful refresh.
Known issues
- On Ignition versions before 8.3.6, some curated metrics are missing because the gateway does not provide them (16 on 8.3.0, down to 9 on 8.3.4). Up to 8.3.3 this includes faulted database connections, so that alert stays silent. Use 8.3.6 or later for full coverage.
- The certificate metric reads the gateway's HTTPS listener on localhost. A gateway whose HTTPS port is bound to one specific network interface reports no certificate series.
- The Settings and Export Status pages use undocumented parts of the gateway web interface, tested on 8.3.6. A later 8.3.x update could break those pages. Export itself would keep working.
- Traces are manual only, through system.observability.span(). Ignition offers no way to trace HTTP requests, database queries or scripts automatically.
- Spans sent straight to Datadog get no APM trace metrics.
- Azure Monitor's OTLP ingestion is still a Microsoft preview.
- Proxy authentication and client certificates (mutual TLS) are not supported yet.
- The curated metrics do not yet cover SQL Bridge, EAM, Sepasoft or the remaining Cirrus modules.
v0.7.0
v0.7.021 Aug 2026OpenTelemetry and Prometheus observability for Ignition 8.3: curated gateway metrics, logs, traces and opt-in audit export, pushed to any OTLP backend or scraped on :9464.
- Ignition:
- 8.3.0+
- Backend:
- Any OpenTelemetry HTTP/protobuf endpoint, or Prometheus scraping the gateway
- Other:
- Ignition Edge: contact Mustry Solutions before deploying
Changes
- Licensing change: the free Community tier is gone. The module is now one paid product, licensed per Ignition Gateway, and everything it does runs under an activation or a live trial. That covers OTLP push, the Prometheus endpoint, curated metrics, the scripting API and health tags. With no licence in effect the module stays installed and keeps every setting, but produces no telemetry: :9464/metrics stops listening, the Gateway route /data/mustry-observability/metrics answers HTTP 503, and curated metrics stop being collected.
- system.observability.* calls become no-ops rather than errors when no licence is in effect, so timer scripts and tag change scripts that publish custom metrics keep running instead of failing the moment a trial lapses.
- Health tags keep publishing whatever the licence state, and two new ones, License/Valid and License/Mode, let you alarm on an expired trial. The Gateway log gets one WARN line naming the state.
- The EULA text changed, so the Gateway will ask you to accept it again when you install or upgrade. Acceptance is stored as a checksum of the licence file, so any edit to the text prompts once more. Containers that set ACCEPT_MODULE_LICENSES are unaffected.
- Rotating a Referenced secret now takes effect when you press Save. The module used to re-read a Secret Provider only when the settings record itself changed, so the obvious gesture after rotating a token centrally, open Settings and Save, did nothing and the old token stayed in force until you edited an unrelated setting or restarted the Gateway. The same gap broke the documented recovery from a fail-closed secret, leaving the Prometheus endpoint down after a fix-and-Save. Unchanged secrets still skip the rebuild, so scans and startup are unaffected, and a rotation is never picked up mid-flight, so a running scrape cannot be broken by a rotation happening elsewhere.
- API change: the license.enterpriseActive field in /status and /health is now license.licensed. The Settings page License row now says "telemetry off" instead of "OTLP export paused".
Known issues
- Direct SaaS push is verified at the wire level, with headers and tokens asserted against capture servers, but not yet against live vendor accounts. The same applies to the Azure Monitor path: the client secret resolves like any other Referenced secret, but no live Azure tenant has been exercised.
- The Settings and Export Status pages are built on undocumented Gateway web UI internals, tested against 8.3.6. A future 8.3.x update could break how those pages render. Telemetry export itself would keep working.
- Traces are beta and manual only, through system.observability.span(). Automatic instrumentation of HTTP requests, database queries and script executions is not available, because the platform offers no injection points for it. The module probes for them at startup and logs what it finds.
- The curated metric packs do not yet cover SQL Bridge, EAM, Sepasoft or the remaining Cirrus modules.