Nobody ever got a push notification from us.

Our Web Push had a settings toggle, a green “On for this browser”, and a zero percent delivery rate. Here are the two bugs — and the reason neither one ever surfaced.

I was auditing our notification system recently, reading through lib/push-notifications.ts, when something stopped me. We pull the subscription keys out of the database:

const result = await query(
  `SELECT id, endpoint, p256dh_key AS p256dh, auth_key AS auth
     FROM push_subscriptions WHERE user_id = ?`,
  [userId]
);

And then we never use them.

Not “use them incorrectly.” Never reference them again. The next thing the function does is stringify the payload and POST it:

const body = JSON.stringify(payload);
// ...
const res = await fetch(sub.endpoint, {
  method: "POST",
  headers,  // Authorization + Content-Type: application/octet-stream
  body,
});

A plain JSON string, sent with a content type claiming it's binary.

If you've worked with Web Push you already know what's wrong. If you haven't: the payload of a Web Push message is required to be encrypted end to end, with a key derived from exactly those two values we were throwing away. The push service — Google's, Mozilla's, Apple's — can't read the message and isn't supposed to. It's a dumb pipe holding an opaque blob for a browser that might be offline. Hand it unencrypted JSON with no Content-Encoding header and it rejects the request.

So our push notifications had never worked. Not degraded. Not flaky. Zero delivered, since the day the feature shipped.

The part that actually bothers me

The bug is embarrassing, but it's a bug. The interesting question is how it survived.

We had a settings page with a toggle. Turning it on fired a real browser permission prompt, got a real PushSubscription, and stored real keys in a real table. The switch then read “On for this browser,” which was completely true. It just didn't mean anything.

And when we sent, we handled the response like this:

if (res.status === 410 || res.status === 404) {
  expiredEndpoints.push(sub.endpoint);
}
// ...
} catch {
  // Network error — ignore
}

Two named outcomes. Everything else falls out the bottom.

Read that carefully. There are exactly two outcomes this code can distinguish: “the subscription is gone,” and “something threw.” Everything else — 200, 201, 400, 401, 403, 429, 500 — falls through the bottom of the function identically. And the function returned void.

We were getting a 400 on every single send. The code had no branch capable of telling a 400 from a 201.

That's the real lesson, and it isn't “test your push notifications.” It's that we wrote error handling that assumed success was the default case. The only paths we named were the two ways we expected things to go wrong. Success was the unhandled remainder.

A function that returns nothing and only inspects the failures it anticipated cannot report a 100% failure rate. It has no vocabulary for the difference.

The one that would have been fatal on its own

While rewriting the encryption I found the second one. Web Push authenticates the sender using VAPID: a JWT signed ES256, carrying the push service's origin as its audience. We were signing it like this:

const sig = crypto.createSign("SHA256")
  .update(signingInput)
  .sign(privateKeyPem, "base64url");

Which is wrong in a way that's genuinely easy to miss, because it produces a perfectly valid ECDSA signature. Node's default output for ECDSA is DER — an ASN.1 SEQUENCE of two INTEGERs, variable length, usually 70 to 72 bytes. ES256 as specified by JWS requires the raw concatenation of R and S: exactly 64 bytes, no wrapper.

The fix is one option:

.sign({ key, dsaEncoding: "ieee-p1363" })

So even if we had been encrypting correctly, every request would have been rejected for a malformed token. Two independent, individually fatal bugs in one feature — and our instrumentation could distinguish neither of them from working.

There was also a createSign object that got constructed, updated, and then never used. Harmless, but it's the kind of thing that tells you nobody had read this function closely in a long while.

Writing it properly

The obvious move is npm i web-push and delete our code. We couldn't: our build environment can't currently reach the npm registry — a CA bundle problem that's its own separate post — so adding a dependency wasn't on the table.

So: hand-rolled, from RFC 8188 (the aes128gcm content coding), RFC 8291 (message encryption for Web Push), and RFC 8292 (VAPID). Which is a sentence that should make you nervous. It made me nervous.

The algorithm itself is short. Generate an ephemeral P-256 keypair. ECDH against the subscriber's public key for a shared secret. Run that through HKDF with the subscription's auth secret and an info string binding both public keys — which is what stops a message being replayed against a different subscription. Then a random 16-byte salt, HKDF again for a content encryption key and a nonce, and AES-128-GCM the plaintext.

const ecdh = crypto.createECDH("prime256v1");
ecdh.generateKeys();
const sharedSecret = ecdh.computeSecret(uaPublic);

const keyInfo = Buffer.concat([
  Buffer.from("WebPush: info\0"), uaPublic, ecdh.getPublicKey(),
]);
const ikm = hkdf(authSecret, sharedSecret, keyInfo, 32);

const salt  = crypto.randomBytes(16);
const cek   = hkdf(salt, ikm, Buffer.from("Content-Encoding: aes128gcm\0"), 16);
const nonce = hkdf(salt, ikm, Buffer.from("Content-Encoding: nonce\0"), 12);

The plaintext gets a trailing 0x02 byte before encryption — the record delimiter marking this as the final record. Leave it off and everything still looks fine: the push service accepts your request, returns 201, and the browser silently discards the message when decryption yields no valid delimiter. That's the nastiest failure mode in the whole spec, because it's the one that looks like success from the server's side.

Then the body is a header, followed by the ciphertext:

Testing crypto you can't send

Here's the part I'm actually pleased with.

You cannot test Web Push against a push service in CI. There's no test endpoint, the real ones need a genuine subscription from a genuine browser, and — as we've just established — a 201 doesn't tell you the browser could read it.

But you don't need one. The push service is a pipe. The thing that has to be able to decrypt your payload is a browser holding a P-256 private key, and you can simply be that browser:

const ua = crypto.createECDH("prime256v1");
ua.generateKeys();
const authSecret = crypto.randomBytes(16);

const body = encryptPushPayload(message, {
  endpoint: "https://example.invalid/x",
  p256dh: ua.getPublicKey().toString("base64url"),
  auth: authSecret.toString("base64url"),
});

// ...then parse the header and decrypt exactly as a browser would.

Then assert you get your plaintext back, that the delimiter is 0x02, and that two consecutive sends don't reuse the ephemeral key. Same trick for VAPID: sign a token, then verify it with crypto.verify(…, { dsaEncoding: "ieee-p1363" }). If the signature is DER, the verify fails and so does the test.

That's in our verification suite now. No network, no database:

ok VAPID self-test passes
ok REGRESSION: a browser can decrypt the payload
ok final-record delimiter is 0x02
ok each send uses a fresh ephemeral key
ok REGRESSION: signature is raw R||S (64 bytes), not DER
ok REGRESSION: the signature actually verifies
ok aud is the push service ORIGIN, not the full endpoint
ok an oversized payload is rejected rather than truncated

Every line marked REGRESSION is a bug we actually had.

What I'd take from this

Two things, and neither of them is about cryptography.

• A feature whose success is invisible needs an explicit success signal. Our send function returned void. It now returns the number of devices that actually accepted the message, and the caller logs anything that isn't a 2xx with its status and response body. That change on its own would have surfaced all of this within a day.
• Be suspicious of code that only handles the errors you predicted. A pair of status checks plus a bare catch {} isn't error handling — it's two guesses, with everything else routed to silence. When the set of outcomes you explicitly name doesn't include the successful ones, you've built something that structurally cannot tell you it's broken.

The toggle works now. If you turned push notifications on at some point and quietly wondered why you never heard from us — that was us. It's fixed.

Part of a broader audit of OneCeylon's notification system, which also turned up an ENUM restated by hand across seven migrations that disagreed with each other, and twenty-five call sites sending booking alerts under the notification type verification. Those are their own posts.