Webhook ingestion operations

Hephaestus receives webhooks in a container of its own and buffers every delivery on NATS JetStream before anything else reads it. Push events on GitHub and GitLab are not redeliverable — the provider's "redeliver" button does not apply to them — so a delivery that is dropped between the receiver and the consumer is gone.

This page is what to watch, what to set, and what to do when it goes wrong. Why it is built this way is ADR 0008.

What to alert on

Metrics are Micrometer, exposed on the receiver's actuator endpoint, all tagged stream with one of github, gitlab, slack, outline.

Metric	Type	What it means
webhook.stream.unacknowledged.deletions	counter	Deliveries lost. A message the stream discarded before the consumer that needed it had read it. Zero unless something is wrong, and irrecoverable when it is not.
webhook.stream.poll.age	gauge (s)	Seconds since the loss counter was last maintained. NaN means never.
webhook.stream.unacknowledged.gap	gauge	How far the worst consumer is behind the oldest message still stored. Rising means a consumer is falling behind while the stream is discarding.
webhook.stream.oldest.message.age	gauge (s)	Your effective retention. The age of the oldest message the stream still holds.
webhook.stream.bytes	gauge	Bytes stored.
webhook.stream.bytes.limit	gauge	The bound in force on the live stream.
webhook.stream.bytes.utilization	gauge (0-1)	bytes over bytes.limit.
webhook.stream.messages	gauge	Messages stored.
webhook.stream.consumers	gauge	Durable consumers on the stream.

Page on any increase in webhook.stream.unacknowledged.deletions. That is webhook data nobody can get back. Every increment is also logged at ERROR, naming the stream, the count and the consumer.

Alert on webhook.stream.poll.age beside it, at something a few times your hephaestus.webhook.stream.monitor-interval (default 60s). A receiver that cannot reach the broker leaves the loss counter flat at zero, which reads exactly like no loss.

Do not page on webhook.stream.bytes.utilization. A healthy stream at its bound sits near 1.0 forever — that is the bound doing its job, not an incident. It is a dashboard number.

webhook.stream.consumers climbing and never falling means durables are accumulating on a shared broker. See Consumers that outlive their deployment.

How long a delivery is actually kept

Each stream carries two bounds, and which one binds depends on your traffic:

• HEPHAESTUS_WEBHOOK_STREAM_MAX_AGE (default 180d) is the ceiling — the longest a delivery is kept if disk allows.
• HEPHAESTUS_WEBHOOK_STREAM_MAX_BYTES (default 1GB, and HEPHAESTUS_WEBHOOK_STREAM_MAX_BYTES_GITHUB at 10GB) is the floor under it. At the bound the broker deletes the oldest messages to admit new ones.

At low volume nothing approaches the byte bound and you get the full 180 days. At high volume the byte bound recycles the stream long before that. Neither number tells you which you have — read webhook.stream.oldest.message.age for your own deployment and treat that as the answer.

Size the byte bound against what a shed message costs you, not against the age ceiling. Nightly reconciliation re-fetches the last MONITORING_TIMEFRAME days from the provider API, so a delivery shed inside that window is recoverable by other means and one shed outside it is recoverable by nothing. If you raise MONITORING_TIMEFRAME, raise the byte bounds with it.

Streams carrying personal message content — slack and outline — are capped at 72 hours regardless of volume. PostgreSQL is the system of record; an erasure request cannot reach inside a broker stream, so those are transport buffers and nothing more.

The storage budget, and why it is not just a number

NATS_JS_MAX_FILE_BYTES (default 16 GiB) sets two things from one value: the broker's own JetStream budget, and the total the receiver checks its per-stream bounds against. The receiver refuses to start if the four bounds sum above it:

Webhook stream bounds total 21474836480 bytes, over the 17179869184-byte broker storage budget
(hephaestus.webhook.stream.storage-budget). Lower hephaestus.webhook.stream.max-bytes[-by-stream],
or raise the budget and the JetStream volume with it.

Keep the budget below the free space on the broker's volume. The two failure modes are not alike:

The broker fills…	What happens
its own JetStream budget	refuses new messages, keeps running, recovers by itself when messages expire
the filesystem	cannot write its own metadata, and stays wedged even after space is freed

The second one needs a human. That is what the arithmetic above exists to make unreachable — for webhook stream growth. max_file governs the whole JetStream store, so anything else sharing that volume can still fill it.

Changing a bound on a stream that already exists

Startup reconciles an existing stream's limits with your configuration. It never rewrites subjects, retention mode, storage type or discard policy.

Situation	Startup does
The change cannot delete anything	applies it, logs at INFO
The change would delete stored messages	withholds it, logs at ERROR with exactly what it would cost
…and HEPHAESTUS_WEBHOOK_STREAM_ALLOW_DESTRUCTIVE_LIMIT_UPDATES=true	applies it, logs what was deleted
The change would leave the stream with no bound at all	withholds it, logs at ERROR — no flag overrides this
Stream state cannot be read	withholds everything; nothing can be proven safe
The live stream's shape has drifted	withholds everything, logs at ERROR

A withheld change looks like this:

Stream github limit change withheld because it would delete stored messages:
[maxBytes -1 -> 10737418240 (32300000000 bytes stored, 21562581760 would be deleted)] —
set hephaestus.webhook.stream.allow-destructive-limit-updates=true to apply it

To apply it deliberately:

1. Run nats stream report and decide whether the size is legitimate for your traffic.
2. If it is, raise HEPHAESTUS_WEBHOOK_STREAM_MAX_BYTES_GITHUB (or ..._MAX_BYTES) instead, keeping the four bounds under NATS_JS_MAX_FILE_BYTES.
3. If it is not, set HEPHAESTUS_WEBHOOK_STREAM_ALLOW_DESTRUCTIVE_LIMIT_UPDATES=true, restart the receiver once, confirm the DELETED stored messages log line, and unset it again.

Leaving that flag set turns a one-time decision into standing permission for any future deploy to delete stored messages.

Shedding the excess happens before the broker replies, so on a large stream the call takes as long as the deletion does. HEPHAESTUS_WEBHOOK_STREAM_LIMIT_UPDATE_TIMEOUT (default 5m) is how long startup waits; raise it if your stream is too large to shed inside that. If it still runs out, startup reports the limits the stream actually has rather than assuming its own update failed — which matters, because a timed-out update is one the broker may well have applied:

Failed to reconcile JetStream stream limits: name=github changed=[maxBytes -1 -> 10737418240]
— live configuration is now maxAge=PT4320H maxBytes=10737418240 maxMessages=-1

That line says the change did land. Confirm with nats stream info github before retrying.

If instead you see a shape-drift error, the stream is not what this deployment expects and no bound is written onto it at all:

Stream github has drifted from the shape this deployment expects: [discardPolicy=New (expected Old)]
— limits left unreconciled

Repair it with nats stream edit github before the bound can be applied. discardPolicy=New in particular makes the broker reject incoming publishes at the bound instead of shedding old messages, which turns a full stream into a total ingestion outage.

Recovering a wedged broker

Symptoms: the receiver's readiness probe is DOWN, webhook.stream.poll.age is climbing or NaN, and the NATS container logs write errors. docker compose ps shows nats-server unhealthy but running — Docker does not act on a failing healthcheck, so it will sit there indefinitely.

1. Confirm it is the disk. On the broker host:

   df -h                                    # free space on the JetStream volume
   docker compose logs --tail=100 nats-server

   no space left on device on a JetStream write is the wedge. If there is free space, this is a different fault — check the network path and nats server check connection instead.

2. Free space. Remove whatever else is on that volume. JetStream will not recover on its own once it has failed to write its own metadata, so expect the errors to continue.

3. Restart the broker, which is the step that clears the wedge:

   docker compose restart nats-server

4. Restart the receiver so it re-establishes its connection and re-reconciles the streams:

   docker compose restart webhook-server

5. Fix the cause before the next incident: set NATS_JS_MAX_FILE_BYTES below the volume's real free space, and bring the per-stream bounds under it. Startup will refuse to run if they do not fit, which is the point.

6. Account for what was lost. Deliveries that arrived while the broker could not write were dropped at the edge; the provider saw a 503. Anything GitHub or GitLab retries comes back on its own schedule — push events do not. Run a reconciliation sync from the instance admin console to re-fetch what falls inside MONITORING_TIMEFRAME; anything older than that window is gone.

Payload size

The receiver accepts webhook deliveries up to 25 MiB, matching GitHub's own delivery cap. The broker must be configured to carry at least that much: NATS_MAX_PAYLOAD_BYTES (default 26214400) sets NATS' max_payload, whose own default of 1 MB is far below it. Every delivery in the gap would be verified, accepted, and then lost at publish.

The receiver reads the broker's live limit at startup and logs at ERROR if the two disagree:

The receiver accepts webhooks up to 26214400 bytes but this NATS server caps a message at 1048576

Consumers that outlive their deployment

Nothing but JetStream removes a durable consumer, and a stack that is deleted rather than shut down never removes the ones it created. On a shared broker that leaves one generation of durables — each with a backlog that will never drain — per stack that ever existed.

HEPHAESTUS_INTEGRATION_CONSUMER_INACTIVE_THRESHOLD (default 30d) lets JetStream reap them. It measures time with nothing connected, not time without traffic: a running deployment holds standing requests against its consumers and resets the clock continuously even while its queues are silent. Only a deployment that no longer exists ages out.

• Set a much shorter value (72h) on disposable stacks that share a broker — preview environments, throwaway test stacks.
• Set 0s to switch reaping off, if your deployment may be offline for more than the threshold and must resume exactly where it left off. A reaped consumer is recreated pointing at new messages only, so it skips whatever arrived while the deployment was gone.
• Values between 0s and 1h are rejected at startup — a threshold that short expires a consumer across an ordinary restart, which is the loss the setting exists to prevent.

0s preserves the consumer's position, not the messages. Those still expire on the stream's own retention, so a deployment offline past the stream's byte or age bound comes back to a cursor pointing at messages the stream no longer holds — webhook.stream.unacknowledged.deletions will say so on the first poll.

Making a change actually reach the broker

Two things in the deployment path will silently leave your change unapplied.

The webhook receiver and NATS are in the core stack, and staging CD deploys the app only. A change to hephaestus.webhook.stream.* ships with the application server but does not reach the container that bootstraps the streams until core is deployed. CI warns when docker/compose.core.yaml changes; deploy it with Deploy to Staging and deploy-core: true.

Compose does not recreate NATS when only its config changes. The broker's configuration is a Compose configs: block whose content renders to a bind-mounted file. Changing the content does not change the container's spec, so docker compose up -d reports no drift and the running broker keeps the config it started with — max_payload and max_file included. Force it:

docker compose -f compose.core.yaml up -d --force-recreate nats-server
docker exec core-nats-server-1 cat /etc/nats/nats-server.conf   # confirm the value you set

Recreating the broker is safe for stored messages — JetStream data lives in the volume — but in-flight deliveries during the restart are lost, and webhooks are not redeliverable. Reconciliation covers MONITORING_TIMEFRAME days afterwards.

Where these settings live

Stream bounds are read by beans that only exist on the webhook runtime role. Setting them on the application-server container configures nothing at all — see Runtime Roles. The reference Compose files put them on webhook-server, and a build-time check fails if they drift back.