AIOps is machine learning applied to operational data: thresholds that adapt to a traffic pattern, correlation of events sharing a single cause, and grouping of alerts into a single ticket. Whether it helps is a readiness question rather than a product question, so this article works through the preconditions in order.
Quick Overview
What you’ll learn:
- Which signal classes the model needs before it can resolve anything
- Why attribute semantics, not the model, decide the quality of correlation
- What dynamic thresholds catch well and what they structurally cannot see
- Where automated response stops and an on-call decision begins
Who this article is for:
- SRE engineers and teams operating production services
- Team leads evaluating AIOps tooling
- People accountable for on-call quality and overnight page volume
Reading time: 7 minutes
What the Model Receives
The signals documentation on opentelemetry.io names the input classes, of which three matter most here: metrics, aggregated numeric measurements over time; logs, records of individual events with context; and traces, linked records of a request moving through services. That source lists further signals and keeps adding to them, but an AIOps layer works chiefly on these. Each answers a different question — a metric says something is wrong, a log says what happened, a trace says where.
A model fed only metrics can point at the moment of an anomaly but cannot attribute it to a service in the call chain, because that information is absent from its input. This is why AIOps rollouts usually stall on missing distributed tracing rather than on the model.
The monitoring chapter in sre.google supplies a readiness bar independent of any tool: latency, traffic, errors and saturation are the signals to start from. That list doubles as a test — if one of them is not collected for a service, buying a tool will not repair the gap.
Attribute Semantics Decide the Outcome
This is the least spectacular part and the one projects lose on. A model correlating events has to know that service.name in one stream and app in another denote the same thing; absent that, it builds independent pictures of a single service and correlates neither.
The semantic conventions on opentelemetry.io address it with agreed attribute names and meanings for services, resources, operations and errors. The payoff is prosaic: only with a shared vocabulary can you ask for all requests belonging to one service regardless of which team instrumented which part. A rollout that opens with tool selection instead of vocabulary buys itself manual field mapping on every new stream.
The practical ordering is therefore the inverse of the intuitive one: instrumentation with agreed attributes first, collection of all signal classes into a common place second, the AIOps layer last. Reversed, you get a deployment where the model runs and its output is not used on call.
What Dynamic Thresholds Catch and Miss
Adaptive thresholds are the most mature application. The documentation on learn.microsoft.com describes dynamic thresholds in metric alert rules as learning a metric’s historical behaviour, deriving an expected range from it and alerting on deviation from the pattern, accounting for daily and weekly seasonality. The benefit needs no promises attached: a rule stops requiring a hand-set value per metric, and a quiet night stops looking like an outage.
The blind spot follows from the mechanism. A state that was bad for the whole observation history becomes the learned normal, so a service that has always responded too slowly will not be flagged. Adaptive thresholds therefore do not replace thresholds derived from a service level objective — those express a requirement rather than a pattern, and both are needed.
Correlation and grouping are the second application. A single database failure pages every service that depends on it; a correlating layer joins those alerts into one ticket on a shared time window and shared resource attributes. The value is that the on-call engineer receives one ticket with a list of effects instead of a list of effects with no cause. Here too the attribute vocabulary decides the quality, not the model.
Where Automation Stops
The boundary is worth writing down before the rollout rather than during an incident. Automation performs actions that are reversible and described in advance — restarting an instance, draining a node from the load balancer, raising a replica count. Irreversible actions, and anything absent from that list, remain a human decision even when the model is confident.
The service owner answers for the service level objectives, because the business requirement lives with them; without objectives the AIOps layer optimises quiet in the alerting channel rather than availability. Whoever maintains the telemetry layer answers for the attribute vocabulary and for new streams arriving with correct names rather than with an ad-hoc mapping.
The postmortem chapter in sre.google adds the part that tooling cannot supply: blameless postmortems exist so that an incident produces a written, actionable record of causes and follow-up. A correlating layer shortens the path to the cause; it does not write the record, and it does not decide which follow-up is worth doing. An organisation without that habit gets faster alerts and the same recurring incidents.
The last thing to state plainly: AIOps does not reduce the need to understand the system. It changes on-call work from reading many alerts to assessing one conclusion together with the evidence behind it — and assessing a conclusion takes more system knowledge, not less.
Read Also
- Observability: logs, metrics and traces in practice — the signal classes and how they are collected
- Observability vs monitoring: a practical SRE guide — the distinction that decides what the model receives
- Blue-green and canary deployments in practice — the first consumer of version-aware metrics
Build Your Skills
This topic is covered by AIOps — automate and optimise IT operations using AI and Observability — OpenTelemetry, Grafana Stack and AIOps. The first covers the model layer and its operational uses; the second covers the telemetry layer without which the first has no input. Talk to us about matching the scope to the state of your monitoring.
Frequently Asked Questions (FAQ)
How does AIOps differ from ordinary monitoring with alerts?
Monitoring compares a measurement against a threshold a person set. AIOps adds a layer that derives the expected range from the metric’s history and joins events sharing a cause into a single ticket. It does not replace thresholds derived from a service level objective — those describe a requirement rather than a behavioural pattern.
Can AIOps be deployed without distributed tracing?
It can be switched on, but its output will be limited to pointing at the moment of an anomaly. Without traces the model has no record of a request crossing services, so it cannot attribute the anomaly to a place in the call chain. That is the most common shape of a deployment that works and goes unused on call.
Should the tool be allowed to act on incidents by itself?
Within reversible, pre-described actions — instance restart, draining a node, changing a replica count. Irreversible actions and anything off that list stay with the on-call engineer. Write the boundary down with the list, because a boundary settled mid-incident always moves toward more automation.
Where do we start with several monitoring tools and no shared vocabulary?
By agreeing attribute names for service, resource and operation, then consolidating streams into a common place. Only with a shared vocabulary is a tool comparison meaningful, because output quality depends on the input more than on the model. The reverse order ends in field mapping on every new stream.