Verify a Sentinel webhook signature
Every message Sentinel posts to your notification destination carries an HMAC-SHA256 of its own body in X-Sentinel-Signature and the time it was signed in X-Sentinel-Timestamp. Below are two complete receivers — one Flask, one Express — that check both and refuse anything that does not add up.
A signature nobody checks is worth nothing. Slack, Discord and Mattermost ignore both headers entirely; for those destinations the secrecy of the URL is the only protection, and that is the model they are built on. This page is for an endpoint you wrote yourself. And checking is only half of it: a receiver that computes the signature and then handles the message anyway has performed a calculation, not a security check. It has to reject. Sentinel cannot do that part for you and cannot tell whether you have.
What these two programs have actually been through
They are not illustrations. Sentinel's test suite starts both of them as real servers, sends them messages signed by the same code that signs yours, and then attacks them: a body with one field edited, a timestamp moved an hour, a message signed with the wrong key, a signature lifted from a different message, a missing header, a future version label, a re-spelt timestamp, and a genuinely signed message replayed two hours later. Each one has to be refused, in both languages.
Two deliberately broken receivers are run through the same battery, because a test that only ever asserts acceptance would pass against a receiver you should not deploy. One hashes the re-serialised JSON instead of the raw body and must reject a genuine message; the other skips the clock check and must accept the replay. Those two are the reason the rules on this page are worth stating.
The key
Each workspace has its own signing key. It appears on that workspace's settings page once a notification destination is set, and both examples read it from the environment:
export SENTINEL_SIGNING_KEY=... # from your settings page
Treat it like a password. Anyone holding it can sign a message your receiver will believe. Replacing it from the settings page takes effect on the next message with no overlap period, so a receiver still holding the old key will reject everything until you update it — which is the receiver behaving correctly.
The rule that catches most people
The signature covers the exact bytes that arrived. A parsed object re-serialised is a different byte string — Python writes {"a": 1} where Node writes {"a":1}, and Python escapes a non-ASCII character that Sentinel sent as UTF-8. Verify against that and every message fails, including the real ones. The bad outcome is not the failure; it is that a check which rejects genuine traffic gets deleted rather than fixed.
The second rule is the clock. A captured message replayed tomorrow is byte-identical and its signature is genuine, so the signature cannot refuse it. The timestamp is inside the signed bytes precisely so that it cannot be edited to now, which is what makes rejecting an old one a real defence. Choose your own tolerance — the examples use five minutes. Sentinel cannot enforce a window at your end.
Python, with Flask
pip install flask · download flask_receiver.py
#!/usr/bin/env python3
"""A complete Sentinel webhook receiver, in Flask.
Copy this file, set SENTINEL_SIGNING_KEY, run it, point your workspace's
notification destination at http://your-host/sentinel.
export SENTINEL_SIGNING_KEY=... # from your Sentinel settings page
pip install flask
python flask_receiver.py
This file is not an illustration. Sentinel's own test suite starts this exact
program, sends it messages signed by the same code that signs yours, and then
sends it a battery of forgeries and replays and asserts it refuses every one.
If you change it, the part most worth re-testing is `verify` -- everything
below it is ordinary web plumbing.
The two mistakes this file exists to stop you making:
1. Verifying the signature and then handling the message anyway. A receiver
that computes an HMAC and does not REJECT on mismatch has performed a
calculation, not a security check. The `return` on the refusal path is the
entire feature.
2. Signing a re-serialised copy of the body. `request.get_json()` gives you a
dict; `json.dumps` of that dict is a DIFFERENT byte string from the one
that arrived -- different spacing, and different escaping for any
non-ASCII character, of which incident titles have plenty. The signature
is over the bytes on the wire and nothing else, so this file reads
`request.get_data()` and parses afterwards.
"""
from __future__ import annotations
import hashlib
import hmac
import json
import os
import time
from flask import Flask, request
# Read from the environment, never pasted into the file. This is the one value
# that lets anyone forge a message your receiver will believe.
SECRET = os.environ["SENTINEL_SIGNING_KEY"]
SIG_HEADER = "X-Sentinel-Signature"
TS_HEADER = "X-Sentinel-Timestamp"
VERSION = "v1"
# How much clock skew you will tolerate, in seconds. Sentinel cannot enforce
# this for you -- it is your end of the exchange. Five minutes is a normal
# choice: long enough to survive an unsynchronised clock, short enough that a
# captured message is not replayable tomorrow.
TOLERANCE_SECONDS = 300
def verify(secret: str, ts_header: str, sig_header: str, body: bytes,
now: int | None = None) -> str | None:
"""Return None if the message is genuine, or a reason to refuse it.
Returning a reason rather than a bool because the reason belongs in your
log. "Refused a webhook" at 4am with no reason is a support ticket.
"""
if not sig_header or not ts_header:
return "missing signature headers"
# An explicit version check. If Sentinel ever signs a different shape of
# string it will say v2, and a receiver written against v1 must refuse it
# rather than compute a v1 signature over it and quietly disagree.
if not sig_header.startswith(VERSION + "="):
return "unknown signature version"
if not ts_header.lstrip("-").isdigit():
return "malformed timestamp"
# The timestamp is signed as the exact string that was sent, not as a
# number that has been through int() and back. Those differ for " 123" and
# for "0123", and a signature scheme with two spellings of the same input
# is a signature scheme with a hole in it.
signed = VERSION.encode() + b":" + ts_header.encode() + b":" + body
expected = VERSION + "=" + hmac.new(
secret.encode(), signed, hashlib.sha256).hexdigest()
# compare_digest, not ==. A byte-by-byte comparison that returns early
# leaks the correct signature to anyone who can time it.
if not hmac.compare_digest(expected, sig_header):
return "signature mismatch"
# Freshness is checked AFTER the signature, deliberately. Before it, you
# would be making a decision about a header that nothing has authenticated
# yet. After it, the timestamp is known to be the one Sentinel signed --
# which is what makes rejecting an old one a real defence against a
# replayed message rather than a formality.
age = (int(time.time()) if now is None else now) - int(ts_header)
if age > TOLERANCE_SECONDS:
return "timestamp too old"
if age < -TOLERANCE_SECONDS:
return "timestamp too far in the future"
return None
app = Flask(__name__)
@app.post("/sentinel")
def sentinel():
body = request.get_data() # RAW bytes. Not request.get_json().
why = verify(SECRET,
request.headers.get(TS_HEADER, ""),
request.headers.get(SIG_HEADER, ""),
body)
if why is not None:
# Log the reason, never the key and never the expected signature --
# printing what you expected hands a forger the answer.
app.logger.warning("refused a Sentinel webhook: %s", why)
return {"refused": why}, 401
message = json.loads(body)
if message.get("event") == "incident.opened":
incident = message["incident"]
# Sentinel retries a failed delivery, and a retry re-sends the SAME
# bytes with the same timestamp and the same signature. If this handler
# does something you would not want done twice -- paging someone,
# opening a ticket -- key it on incident["id"], which is stable across
# retries. Do not key it on the signature, which is stable too and will
# therefore also suppress a genuine second delivery after a rotation.
handle_incident(incident)
return {"ok": True}, 200
def handle_incident(incident: dict) -> None:
"""Yours to write. This is what the payload contains."""
print(f"incident {incident['id']}: {incident['title']} "
f"({incident['alert_count']} alerts)")
similar = incident.get("similar")
if similar:
# Sentinel found an older incident of yours that looks like this one.
# `action_item` is a sentence one of your team typed at the time.
# It is not a diagnosis and Sentinel does not claim it is one.
print(f" similar to {similar['date']}: {similar['action_item']}")
else:
print(" no similar resolved incident in this workspace's history")
if __name__ == "__main__":
app.run(host="127.0.0.1", port=int(os.environ.get("PORT", "8080")))
Node, with Express
npm install express · download express_receiver.js
#!/usr/bin/env node
/**
* A complete Sentinel webhook receiver, in Express.
*
* Copy this file, set SENTINEL_SIGNING_KEY, run it, point your workspace's
* notification destination at http://your-host/sentinel.
*
* export SENTINEL_SIGNING_KEY=... # from your Sentinel settings page
* npm install express
* node express_receiver.js
*
* This file is not an illustration. Sentinel's own test suite starts this
* exact program, sends it messages signed by the same code that signs yours,
* and then sends it a battery of forgeries and replays and asserts it refuses
* every one. If you change it, the part most worth re-testing is `verify` --
* everything below it is ordinary plumbing.
*
* The mistake this file exists to stop you making is `express.json()`.
*
* The signature is over the bytes on the wire. `express.json()` consumes those
* bytes and hands you a parsed object, and `JSON.stringify` of that object is
* a DIFFERENT byte string: Node writes {"a":1} where the sender wrote
* {"a": 1}. Every signature check against it fails, for every message, and the
* obvious way out of a check that fails on genuine traffic is to stop
* checking. So this route uses express.raw() and parses afterwards.
*
* If something else in your app has already installed express.json() globally,
* the raw bytes are gone by the time your handler runs. Either mount this
* route before that middleware, or capture the body as it arrives:
*
* app.use(express.json({
* verify: (req, res, buf) => { req.rawBody = buf },
* }))
*/
const crypto = require("node:crypto");
const express = require("express");
// Read from the environment, never pasted into the file. This is the one value
// that lets anyone forge a message your receiver will believe.
const SECRET = process.env.SENTINEL_SIGNING_KEY;
if (!SECRET) throw new Error("SENTINEL_SIGNING_KEY is not set");
const SIG_HEADER = "x-sentinel-signature";
const TS_HEADER = "x-sentinel-timestamp";
const VERSION = "v1";
// How much clock skew you will tolerate, in seconds. Sentinel cannot enforce
// this for you -- it is your end of the exchange. Five minutes is a normal
// choice: long enough to survive an unsynchronised clock, short enough that a
// captured message is not replayable tomorrow.
const TOLERANCE_SECONDS = 300;
/**
* Returns null if the message is genuine, or a reason to refuse it.
* The reason is for your log; "refused a webhook" with no reason is a support
* ticket at 4am.
*
* @param {string} secret
* @param {string} tsHeader
* @param {string} sigHeader
* @param {Buffer} body the RAW request body, not a re-serialised object
* @param {number} [now] unix seconds; defaults to the clock
*/
function verify(secret, tsHeader, sigHeader, body, now) {
if (!sigHeader || !tsHeader) return "missing signature headers";
// An explicit version check. If Sentinel ever signs a different shape of
// string it will say v2, and a receiver written against v1 must refuse it
// rather than compute a v1 signature over it and quietly disagree.
if (!sigHeader.startsWith(VERSION + "=")) return "unknown signature version";
if (!/^-?[0-9]+$/.test(tsHeader)) return "malformed timestamp";
// The timestamp is signed as the exact string that was sent, not as a number
// that has been through Number() and back. Those differ for "0123", and a
// signature scheme with two spellings of the same input has a hole in it.
const signed = Buffer.concat([
Buffer.from(`${VERSION}:${tsHeader}:`, "utf8"),
body,
]);
const expected =
VERSION + "=" +
crypto.createHmac("sha256", secret).update(signed).digest("hex");
// timingSafeEqual, not ===. A comparison that returns early leaks the
// correct signature to anyone who can measure it. It throws on a length
// mismatch, so the lengths are compared first -- that comparison reveals
// only the length, which is fixed and public anyway.
const a = Buffer.from(expected, "utf8");
const b = Buffer.from(sigHeader, "utf8");
if (a.length !== b.length || !crypto.timingSafeEqual(a, b)) {
return "signature mismatch";
}
// Freshness is checked AFTER the signature, deliberately. Before it, you
// would be making a decision about a header nothing has authenticated yet.
// After it, the timestamp is known to be the one Sentinel signed -- which is
// what makes rejecting an old one a real defence against a replayed message
// rather than a formality.
const clock = now === undefined ? Math.floor(Date.now() / 1000) : now;
const age = clock - Number(tsHeader);
if (age > TOLERANCE_SECONDS) return "timestamp too old";
if (age < -TOLERANCE_SECONDS) return "timestamp too far in the future";
return null;
}
const app = express();
app.post("/sentinel", express.raw({ type: "*/*" }), (req, res) => {
const body = req.body; // a Buffer of the RAW bytes, because of express.raw
const why = verify(
SECRET,
req.get(TS_HEADER) || "",
req.get(SIG_HEADER) || "",
body,
);
if (why !== null) {
// Log the reason, never the key and never the expected signature --
// printing what you expected hands a forger the answer.
console.warn(`refused a Sentinel webhook: ${why}`);
return res.status(401).json({ refused: why });
}
const message = JSON.parse(body.toString("utf8"));
if (message.event === "incident.opened") {
// Sentinel retries a failed delivery, and a retry re-sends the SAME bytes
// with the same timestamp and the same signature. If this handler does
// something you would not want done twice -- paging someone, opening a
// ticket -- key it on incident.id, which is stable across retries. Do not
// key it on the signature, which is stable too and will therefore also
// suppress a genuine second delivery after a rotation.
handleIncident(message.incident);
}
return res.status(200).json({ ok: true });
});
/** Yours to write. This is what the payload contains. */
function handleIncident(incident) {
console.log(
`incident ${incident.id}: ${incident.title} ` +
`(${incident.alert_count} alerts)`,
);
if (incident.similar) {
// Sentinel found an older incident of yours that looks like this one.
// `action_item` is a sentence one of your team typed at the time. It is
// not a diagnosis and Sentinel does not claim it is one.
console.log(
` similar to ${incident.similar.date}: ${incident.similar.action_item}`,
);
} else {
console.log(" no similar resolved incident in this workspace's history");
}
}
// Exported so a test can call `verify` directly. The server only starts when
// this file is run, not when it is required.
module.exports = { verify, app };
if (require.main === module) {
const port = Number(process.env.PORT || 8080);
app.listen(port, "127.0.0.1", () => {
console.log(`listening on 127.0.0.1:${port}`);
});
}
Questions
Do I have to check the signature?
No. If your destination is a Slack, Discord or Mattermost incoming webhook, those services ignore both headers and there is nothing you can do about it — the secrecy of the destination URL is the only thing protecting it, which is the model those products are built on. The signature is for an endpoint you wrote yourself.
What happens if I verify the parsed JSON instead of the raw body?
Every message fails, including genuine ones. A re-serialised object differs from the bytes on the wire in spacing, and in the escaping of any non-ASCII character — and incident titles contain plenty. The danger is that a check which fails on real traffic gets removed rather than fixed.
Does checking the signature stop a replay?
Not on its own. A captured message is byte-identical to the original and its signature is genuine, so no amount of signature checking will refuse it. The timestamp window is the part that does, which is why the timestamp is inside the signed bytes rather than merely sent alongside them.
Will a retry arrive twice with the same signature?
Yes. Sentinel retries a failed delivery by re-sending the identical bytes, with the same timestamp and the same signature. If your handler does something that must not happen twice, key it on the incident id rather than on the signature.
The signature format is versioned. Today it is v1, and both examples refuse a label they do not recognise rather than checking it as v1 anyway.