How it works, exactly
Every rule below is the rule the code runs. Where a number appears here it is the number in the source, not a rounded description of it.
1 · Intake is a webhook or forwarded email
Each workspace gets one address at in.sentinel.deplyra.com, shown
on the workspace page. Point your alerting tool's email notifications at
it — most of them can send to an arbitrary address, and this needs no
integration, no OAuth app and no access to your infrastructure.
The address carries random bytes rather than being w7@…. It is
a capability: anyone who knows it can write into that workspace's history,
and the entire value of the product is that the history is real.
A workspace can also have a webhook URL, created from its
settings page and not before — a live URL that can write into your incident
history is not something to hold by default. POST to it from Prometheus
Alertmanager or Grafana with no adapting, or from anything else that can
send JSON with a summary, title or
message field.
One Alertmanager POST usually carries a whole batch. Every alert in it becomes its own row and the grouping rule below decides whether they are one incident — the same rule, reading the same alert text, whichever door the alert came through. Alerts from your webhook and your email address land on the same card during the same outage.
Two things it will not do with a payload. A resolved or OK notification is never stored as an alert, because a cleared condition stored as one would open an incident whose only content is the news that something is fine — and would then be matchable text, so a future suggestion could cite a recovery. It is handled separately; see step 3. And a payload it cannot find alert text in is refused with a 422 naming the fields it looked for, rather than stored under a placeholder title. Nothing invented gets into a history whose whole value is that it is real.
The webhook URL is itself the credential — there is no separate password, so it will appear in the logs of whatever proxy sits in front of your alerting tool. It is one click to revoke and one to rotate. It is deliberately not also in our logs: the access log redacts it.
2 · The grouping rule, in full
No model decides that two alerts are the same incident. That decision is the one that silently destroys information — two incidents merged means the second is on nobody's screen — so it is made by a rule you can read and argue with, and the rule that fired is printed on every alert.
- Same fingerprint, within two hours. The subject is normalised — firing counts, timestamps, UUIDs, IP addresses, percentages and bare numbers are replaced with placeholders — and hashed with the service name. The identical check firing again is the same incident.
- Same service, within twenty minutes. This is what collapses a real storm: "connection pool exhausted", "api 5xx rate" and "checkout timing out" are three checks and one outage. The window is much shorter than the first rule on purpose — this one is an inference about different symptoms, and a long window would swallow a genuinely separate incident an hour later.
- Otherwise, a new incident.
Where a service name cannot be found in the alert, it is recorded as unknown and stays unknown. Rule 2 then cannot fire, so two unattributable alerts are never merged on the strength of a guess.
3 · When your tools say it stopped
Alertmanager, Grafana and most other tools send a notification when a condition clears. Sentinel can close the card on it, under a rule as explicit as the grouping one — because a rule that closes an incident is a rule you are entitled to read before it does.
- A clear is never stored as an alert and never opens an incident. It is recorded in its own table, so it can never become text a future incident is matched against.
- It is matched to an open incident by identical fingerprint — the same check — and not by service. A storm card holds several different checks, and the database alert clearing tells you nothing about whether checkout is still timing out.
- The card closes only when every distinct check on it has cleared, each after its own most recent alert. Until then it stays open and says how many checks are still firing. Closing a six-alert card because one check recovered would take a live outage off the screen of the person handling it.
- A check that fires again after clearing has an alert newer than its clear, so the clear stops counting. Flapping needs no separate rule.
- A clear that matches nothing is recorded and does nothing — including whether the incident it named was already resolved, because "your clears are not reaching us" and "they reached us and matched nothing" have different fixes and look identical otherwise.
Over email the rule is narrower still: only a bracketed prefix —
[RESOLVED], [OK], (Resolved) — counts
as a clear. "DNS lookup could not be resolved" is a real alert and treating
it as a recovery would drop it from your history and close a live card. The
cost is that a tool writing RESOLVED: with no brackets is not
recognised, and its clear is stored as an ordinary alert.
4 · The action item is what writes the library
Closing an incident asks one question: what actually fixed it. That sentence is exactly what a future incident will be shown, so it is typed by a person and never generated. A generated action item would make tomorrow's suggestion a suggestion sourced from yesterday's suggestion, with no evidence at the bottom of the chain.
So an incident your alerting tool closed is closed and not in the library. Nobody wrote down what fixed it, so there is nothing to show anyone — and the workspace page counts these separately, because otherwise a team whose tools close their own cards watches resolved incidents pile up while the library stays at zero and concludes the matching is broken.
5 · Matching, and the two thresholds
Matching is Postgres full-text search over incidents in your workspace that are resolved and have an action item written on them. It is lexical, not semantic, and not a trained model. A candidate is shown only when it clears both of:
- a rank score of
0.05, and - at least three distinct shared terms.
The second threshold exists because rank alone rewards a single rare word. One shared hostname would otherwise link two unrelated outages and put a confident number next to the link. Below the bar, the screen says nothing is similar — which is a real answer and the one a new workspace gets for weeks.
Everything shown is sourced. A suggestion carries the prior incident's date, its title, the exact overlapping terms, and the action item as it was written. There is no path in the code by which a suggestion appears without a stored prior incident behind it: the database column that points at it cannot be null.
Where a language model is, and is not, involved
A model never chooses a match and never sees your library. Once search has already linked two incidents, a model may be asked for one sentence about what the two records have in common. That sentence is then checked, outside the code that calls the model:
- any number in it that is not in the two records → dropped;
- any substantive word not in the two records → dropped;
- any statement of cause, any recommendation → dropped.
A dropped sentence costs a little polish. The match is still shown, with its shared terms and no prose. Every model call is recorded with its token counts and cost in tenths of a penny, visible on your settings page.