0 0 0 Subscribe

Satsuma Engineering · Monday, 26 October 2026

The lab

False RPC-down alerts traced to a checker that gave up after one timeout

A fix to Satsuma's status checker was merged in the satsuma-status repository, with tests and CI passing. The checker is a Cloudflare Worker that sends requests to the chain's RPC endpoint, the public entry point that wallets and tools talk to, and raises an alert when it gets no answer. Now each check retries once before it counts as failed, and the RPC unreachable alert needs 3 consecutive failed minutes before it fires.

The session, run with Claude in Claude Code, checked the Friday window from 17:31 to 17:48 UTC, when alerts kept firing and resolving. The separate monitor box saw zero failures all week, with the slowest call at 0.8 s; the validator kept producing blocks; and the Caddy, nginx and RPC logs showed no errors. "Those alerts were false alarms: the chain and RPC were healthy the whole time."

The alerts were treated as false alarms, most likely short network blips between Cloudflare and the servers. That cause is inferred, not confirmed. The new threshold matches the monitor box, which already waits 3 minutes before alerting.

The fix only takes effect once the checker is redeployed. The deploy was blocked during the session and still waits for a person to run it.

explored
  • Traced repeated alert and resolve notifications for the chain's RPC endpoint
  • Checked monitor box probe data, validator block production and server logs for the Friday 17:31–17:48 UTC window
  • Tried to deploy the updated checker; the deploy was blocked
decided
  • The alerts were false alarms, most likely short network blips between Cloudflare and the servers
  • The Worker's alert threshold now matches the monitor box, which already waits 3 minutes
open
  • The checker redeploy is pending and needs a person to run it
  • The cause of the missed requests is inferred, not confirmed