SMAD Picklebot · Build Pipeline

GHA and GCB Dual Pipelines

Last updated 11/15/26 11:21 AM PT

Every pipeline can build and deploy on GitHub Actions or on Google Cloud Build. Pushes run on the preferred path; the other stands by and takes over if the first cannot start. Everything is built once on the PC before the push: the app, and every image the backend runs, the ten functions included since 10/18/26; the PC tests it, the Mac mini checks each iPhone update, and the deploys run on the PC's own runners, with both paths shipping that same build. Every backend component lands on a dev rung first and climbs to production on the owner's word; slow steps run side by side. Here is how, which path runs, what each costs this month, and how fast each gets a push live.

Built once on the PC, checked on the Mac mini

Two machines of our own do the building and the checking, at no cost per run. The PC builds the app from a committed tree only: the typecheck and every app test, the web export, the iPhone bundle and the check that the app's native fingerprint still matches the installed phones all run side by side, and the build is kept with a checksum under the content of the app and its public config. GitHub and Cloud Build both fetch that one build, check it, and ship it, instead of each building its own: what was tested is what goes live, and a failover ships the same bytes. Every push builds the web; the iPhone bundle is built only when a phone update ships, and goes to Expo's dev channel. The Mac mini (a 2018 Intel Mac) runs an iPhone simulator that listens on that dev channel: it downloads the update, and Maestro checks that the sign-in screen draws with this build's stamp. Only an update the Mac mini passed is republished to members' iPhones. A phone update built without the PC says so in its run: a cloud-only build, the emergency path. Twelve-factor's fifth rule: build, release, run.

The two machines also run the tests and the smoke tests, off the billed runners:

The latest releases (web 027a191, iPhone 027a191, backend 8431e74), as the Release Dashboard recorded them, the pipeline's own stages among the machines':

WhereStageTime
PCPython suites this push can break, in the Linux test image1m 3s
PCWeb export39s
PCWeb upload4s
PCWeb build in all, tests and export side by side2m 21s
PCiPhone export, beside the web export1m 19s
PCiPhone update to the dev channel25s
PCWeb and iPhone build in all2m 21s
Mac miniSimulator: waiting for the update to download13s
Mac miniSimulator: Maestro smoke-tests the sign-in screen and stamp25s
Mac miniSmoke test of the iPhone update in all42s
PipelineWeb app live after the push47s
PipelineThe alert mailer: its deploy job, to the dev rung27s
PipelineThe alert watcher: its deploy job, to the dev rung26s
PipelineThe API: its deploy job, to the dev rung32s
PipelineApp Store watch: its deploy job, to the dev rung25s
PipelineThe CI metrics job: its deploy job, to the dev rung28s
Pipelineclaude-runner: its deploy job, to the dev rung16s
PipelineThe court booking job: its deploy job, to the dev rung29s
PipelineThe daily job: its deploy job, to the dev rung37s
PipelineThe event worker: its deploy job, to the dev rung43s
PipelinePicklebot: its deploy job, to the dev rung26s
PipelinePlay Console watch: its deploy job, to the dev rung27s
PipelineThe push decider: its deploy job, to the dev rung36s
PipelineThe WhatsApp sender: its deploy job, to the dev rung27s
PipelineSentry alerts: its deploy job, to the dev rung26s
PipelineThe website: its deploy job, to the dev rung27s
PipelineThe usage checks jobs: its deploy job, to the dev rung34s
PipelineThe Venmo trigger: its deploy job, to the dev rung27s
PipelineThe vote writer: its deploy job, to the dev rung25s
PipelineThe watchdog job: its deploy job, to the dev rung29s
PipelineThe vote webhook: its deploy job, to the dev rung28s
PipelineEvery backend component at the dev rung after the push40s
PCSmoke test of the live web app, signed in19s
PCSmoke test of the deployed backend14s

Steps side by side

Slow steps that do not wait on each other run at the same time. On the PC, the typecheck and tests, the web export, the iPhone bundle and the fingerprint check run together, and each export uploads the moment it is packed while its source maps go to Sentry: 72 s became 56 s on 10/13/26. On the Mac mini, the simulator waits only until the app's own log says the update is in, and Maestro keeps its driver between runs: 71 s became about 40 s. On GitHub, the smoke test's browser and the desktop's build download while the packages install and the tests run, all on the one machine, so the billed minutes do not change; only the wait does. The daily and usage checks jobs deploy together.

One app change, each path started by hand, push to live:

PathPush to liveRunningQueued
GitHub Actions, steps one after another3m 16s3m 6s0s
GitHub Actions, slow downloads beside the tests2m 11s2m 2s0s
GitHub Actions, a web-only push1m 23s1m 23s0s
Cloud Build3m 46s2m 17s1m 7s

Each step, before and after:

StepGitHub beforeCloud Build beforeAfter
Web export34s–53s27s3s–4s
iPhone bundle45s–1m 2s47sonly on a phone deploy, on the desktop
Smoke test1m 13s48s35s–54s

Where things stand

Pushes run on
GitHub Actions
The other path stands by. Cloud Build's standby, the push decider: push-decider on.
GitHub Actions, November 2026
220 min
3,000 included · billed $0.00
Cloud Build, November 2026
47.3 min
2,500 free · billed $0.00, every Cloud Build charge including the 8-CPU pool

Push to live, per pipeline

Each pipeline's newest run on GitHub, then the median from push to live over the last 30 days on each path, the faster median in green; "—" where a path has no runs, or its builds cannot be told apart. The newest run is where a pipeline stands today; a month's median still carries the runs from before the latest change.

PipelineGitHub, newestGitHub, medianrunsCloud Buildruns
App52s52s1082m 35s6
API57s53s771m 42s1
Picklebot48s45s87——
WhatsApp Vote Webhook47s47s65——
WhatsApp Message Sender42s42s59——
Venmo Sync Trigger48s44s59——
Vote Writer44s45s86——
Event Worker56s50s22——
Site40s39s44——
Alert Watcher45s46s46——
Sentry Alerts48s45s57——
App Store Watch46s46s67——
Play Console Watch46s45s66——
Push Decider47s44s88——
Claude Runner46s45s72——
Court Booking Job32s32s62—0
Daily Job1m 16s51s956m 24s1
Usage Checks Job1m 3s39s891m 58s4
Watchdog Job30s31s621m 33s1
CI Metrics Job33s29s6150s1
Tests Image2m 44s2m 42s383m 6s11
Tests——01m 46s8

The architecture

Every push reaches one decision. From there two complete tracks run side by side, each able to finish the whole job alone. The preferred track takes the push; the other stands by. Before the decision, the PC has already tested and built the push; after either track's deploy, the new revision waits on a dev rung until the owner promotes it.

GitHub Actions preferred · runs the push · on the PC's four runners when the PC built it, the API on GitHub's (Docker) Cloud Build standby · takes over if GitHub is silent · the same deploy scripts, the same ladder Decision every push The PC, before the push tests, app build, every image, gated on every push since 10/19 The override, the decider backend=gha on a manual run, a trigger by hand; 1:30 AM's choice Claimall at once, 9 s Deploy gatechanged? else skip Buildthe PC's, found Deploy to devby digest, no traffic Smoke at devor the PC's; held Promotewarmed, on the word Live 2 min to dev Claimwaits up to 75 s Deploy gatethe same decision Buildthe PC's, found Deploy to devthe same script Smoke at devsame checks; held Promotethe same run Live 4–5 min to dev One claim per push in Firestore. The preferred path runs it; the other steps aside, or takes over at 75 s. The PC's build web and phone; 16 images, 10 of them functions; memo'd Held at the dev rung serving at its dev tag until one Promote run moves every held service, about 80 s. The push decider starts Cloud Build only when GitHub never claimed Solid track: GitHub, the preferred path, which claims every push; its jobs run on the PC's own runners when the PC built the push. Dashed track: Cloud Build, idle until the push decider finds a push GitHub never claimed within 75 s. Either track ends at the held dev rung. Order of preference (10/19/26): the PC's own runners, then GitHub's, then Cloud Build, for cost and speed.

What each pipeline builds

One repository, two build paths, nine kinds of system. Each row is what gets built, the tools that build it, and where it runs or ships.

WHAT IT IS BUILT WITH RUNS ON · SHIPS TO GitHub SMADPickleBot repo every push to main starts the pipelines it touches Decision every push GitHub Actions preferred · ~2 min to dev on the PC's runners, when it built Cloud Build standby · 4–5 min to dev own trigger: every pipeline, on ALSO IN EVERY BUILD Python 3.14, one version 16 images built once on the PC deploys only what changed smoked at dev, held, promoted Picklebot /pb commands, vote webhook, sender, vote writer, Venmo, alerts, Sentry, App Store watch Docker image → dev each function's image built once on the PC, deployed by digest to its own service Production: 10 functions held at dev after its smoke, promoted on the owner's word; Pub/Sub, Firestore API FastAPI, Clerk sign-in; what the app and the bot read and write Docker its own Dockerfile; Alembic migrates before each deploy dev tag → Cloud Run + Neon no-traffic revision smoked at its own URL, held, then promoted; Neon Postgres PW Scheduled jobs court booking (with its own browser), daily, usage checks, weekly, watchdog, metrics Docker slim images built once on the desktop; the browser image 1.1 GB → 347 MB Cloud Run Jobs each started by its own Cloud Scheduler job (one per task) smadpicklebot.com the site and its dashboards: DUPR, finance, costs, releases, post-mortems Docker image → dev the site's function climbs the same rung; pages rendered by the jobs that own them Storage + function pages in the reports bucket, served with security headers; Cloud DNS React web app smadpicklebot.com/app: Today's Game, votes, matches, roster, League Desktop build → dev tests and exports side by side, ~56 s, serving on dev before any push Production takes only the build dev is serving; a signed-in robot checks it after iPhone updates React Native, the web app's code, over the air, when the owner says Dev → simulator on Expo's dev channel, checked on the Mac mini's iPhone simulator, ~40 s Members' iPhones on "ship it", the very update the simulator passed; no pass, no update iPhone store builds new native code only; a guard refuses any change that would strand updates Expo EAS started from the PC's runner, Cloud Build standing by; large, cached TestFlight → App Store sent by Expo or the Mac mini's Transporter; Apple's webhook reports Android app future: the same React Native code, held until the owner says go Expo EAS the same Build and Update, an Android target added Google Play internal testing, then review (not started) Infrastructure schedules, DNS, buckets, triggers, permissions, Sentry alerts Terraform a format-checked plan per change, apply on push; a held lock waits Google Cloud project smad-pickleball, us-west1; Sentry's alert rules Tests every suite once a day, and only the ones a push can break per push pytest the PC first, in the Linux image; GitHub or Cloud Build without it Results dashboard results to the reports bucket; a red run files an issue WHEN A BUILD FAILS, IT SAYS SO GitHub issue the error in its title Email one plain message per failure Slack #smad-picklebot, with buttons Sentry by release, the suspect commit

Logos from Simple Icons (CC0), drawn in the page. Google Cloud's own product marks, Slack and TestFlight aren't in that set, so they show as simple labeled shapes. Dashed: the future Android app. The app's web and iPhone pipeline is Black Widow's.

The app's ladder

The web app and the iPhone app climb the same way: built once on the desktop, run on a rung below production, and only that same build moves up. A rung above never runs a newer build than the rung below, and the pipelines refuse one that skipped a rung.

Web app every push · smadpicklebot.com/app Desktop build871 tests, exports sideby side, 56 s, 686 KB Dev rungdev.smadpicklebot.com,9 pages checked, signed in Push to mainrefused for a build thedev rung isn't serving Productionthe same files, live abouta minute after the push Smoke testthe desktop's robot,4 pages, signed in iPhone updates over the air · when the owner says "ship it" · since 10/13/26 · each stage timed on the Release Dashboard Desktop phone bundlebeside the web export,36 s; fingerprint checked Expo dev channelthe same bytes, about 18 s;only the simulator listens Mac mini simulatorMaestro checks sign-in andthis build's stamp, ~40 s "Ship it"the owner's word; the pushrepublishes what passed Members' iPhoneslive ~2 min after the push;applies itself, no tap verdict New native versions rare · new native code · by hand, on the owner's explicit yes Fingerprint guardrefuses changes thatwould strand phones Store buildExpo's large machine, fromthe PC's runner (CloudBuild standing by) TestFlight → App StoreExpo or Transporter sends;listing gaps shown whileExpo builds; Apple's webhook Simulator shellone per native version,installed on the Mac mini The shell is the compiled part; each update rides on the one its version matches. Solid: every release. Dashed: only when the app's native code changes. The thick green step is the gate: no passing verdict, no phone update.

The backend's ladder

Every function, the API, the website and the scheduled jobs climb the same way since 10/18/26: tested and built on the PC before the push, deployed with no traffic to a dev rung and smoke-tested there, then held until the owner says promote. One Promote run moves every held service at once; a job image climbs through a twin job that runs first. The same artifact, the same settings and the same data on every rung.

Services the ten functions and the API · every push that reaches them The PC, beforeits suites, under Docker;each image, by content Push to maingated: tested, built, onmain's tip, or refused Dev rungits dev tag, no traffic;24–46 s a function Smoked, then heldchecks at the tag's URL;a failure never serves Promotethe owner's word; warmat dev, then traffic Productionsame revision, trafficmoved; the PC smokes it held Job images court booking, daily, usage checks, watchdog, CI metrics · each started by its own Cloud Scheduler job The PC's imageslim, what it runs only;found by digest after Push to mainthe job's deploy on aPC runner, 30–60 s The twin runs firstthe -dev twin, executedon the same image first Heldthe scheduled job keepsits image till the word Climbthe owner's word; twin,then the job, ~40 s The scheduled jobits next run on the newimage; a deploy record Every rung is a GitHub deployment record (dev, production), so the repository's Deployments view is the release console. The thick steps are the holds: nothing passes them without the owner's word.

Each step

1 · THE DECIDER

Picks the preferred path daily

The 1:30 AM billing run reads GitHub's minutes and billing and Cloud Build's free tier. It prefers GitHub, the faster one, unless GitHub would be blocked from running, and saves the choice per pipeline in Firestore.

2 · THE OVERRIDE

A manual run, or a trigger by hand the switch

The GitHub Variable CI_BACKEND chose GitHub on 10/9/26; since 10/11/26 the claim decides per push, and GCB_TRIGGERS is the one switch that turned every Cloud Build trigger on and retired the old hand-offs. A manual run picks GitHub with backend=gha; a Cloud Build run by hand is the pipeline's own trigger, started with the commit and no claim.

3 · BOTH TRACKS START

Two independent starts per push

GitHub starts its job from the push. Cloud Build has its own trigger for every pipeline on the same push (seventeen, generated from each workflow's own path filter so the two always start together), with no GitHub job in between, so either cloud can be down without stopping the other.

4 · THE CLAIM

One path takes the push seconds

The preferred path writes one record for the push in Firestore. The Deploy run's first job claims every component the push touches at once, in about nine seconds, so a job waiting its turn for a PC runner can never lose the push to the standby. On the Cloud Build side nothing runs at all: a small function on GitHub's push webhook, the push decider, notes which pipelines the push touches, and a one-minute sweep starts Cloud Build only for a push GitHub never claimed within 75 seconds. A step-aside costs nothing; the gate builds that did this job for a day (13 to 46 billed seconds each, about 115 build-minutes a day) are gone.

5 · THE DEPLOY GATE

Only what changed both

Each pipeline asks whether this push changes what it runs. A function or job that isn't touched is skipped, with the reason in the run summary. The tests run only the suites the push can break, and every suite once a day. Since 10/13/26 each deploy's own path filter is generated from the same import walk, so a push that touches no deploy's code starts no deploy at all, and the tests on GitHub skip a commit the desktop already tested: the desktop runs the suites in the very Linux image Cloud Build uses, on a clean copy of the commit, and leaves a mark the runner reads. Checking the workflow files moved into the tests job, one run fewer per push. Since 10/18/26 the push itself is the gate, and since 10/19/26 for every push, not only the app's: the PC's tests and image build run before the push leaves (a few seconds when their marks already answer), a push that changes the app also needs a fully tested app build, and a failure refuses the push. Any push whose head is not on top of main is refused in a second, before a mark is spent; a main that moves while the gate runs is rebased onto in place, a mark whose inputs the rebase did not touch is carried to the new commit, and the second push clears in seconds. The gate is one shared program the owner's other projects run too, each with its own list of checks.

6 · BUILD ONCE, THEN DEPLOY

One build, shipped by either path both

Everything is built once, on the desktop, from the committed tree, before the push: the web export on every push, the iPhone bundle only when a phone update ships, since 10/11/26 every job image, and since 10/18/26 every function as an image too (Cloud Functions' own deploy had rebuilt each function's container on Cloud Build on every deploy, about a minute and a billed build each; now each function's image holds the Functions Framework and is deployed by digest onto the function's own service, its address, trigger and permissions untouched: a function's deploy step went from 75–103 s to 24–46 s). Each artifact is keyed by its content and marked as the desktop's, and the PC keeps a memo of what it last found or built, so a push whose images already exist clears the build gate in about a minute with no registry or bucket call. Both paths find that build, check its digest, and deploy it with the same deploy script, so the two can't drift apart and a failover ships the same bytes; a push the desktop did not build is built once by whichever path gets there first. Images ship only what they run (the browser images went from 1.1 GB to 347 MB and 399 MB), every deploy log prints the build's size beside the last one's, and the release is marked in Sentry once per push (since 10/13/26: one push had sent ten "deployed" emails, one per function). The API migrates its database before the new version deploys.

6½ · ONE SNAPSHOT OF SETTINGS

The same settings on either path both

A pipeline names its settings once, in its workflow: GitHub's Variables and a few group ids. Every deploy job writes them into one snapshot in Secret Manager, and a Cloud Build build renders its workflow's settings from that snapshot, so a setting changed on GitHub reaches either path and the two cannot deploy different values.

7 · SIDE BY SIDE

Slow steps run in parallel both

On the desktop, the app's typecheck and tests, the web export, the iPhone bundle and the native fingerprint check all run at the same time, and each export uploads the moment it's packed while its source maps go to Sentry: the app build went from 72 s to 56 s on 10/13/26. It is limited by the processor alone (the minifier is most of it), never by disk or memory. On GitHub, the smoke test's browser and the desktop's build download while the packages install and the tests run, on the same machine, so the billed minutes don't change. The daily and nightly jobs deploy together.

7½ · WHERE THE JOBS RUN

The PC's own runners, free GitHub

Since 10/15/26 a push the PC built deploys on GitHub Actions runners that run on the PC itself, four of them in containers, side by side and at no billed minute; the Deploy run's first job picks them when they are online and the commit carries the PC's build mark. The API's job stays on GitHub's runner because it starts its new image and runs the database migration inside it under Docker, which the PC's runner containers do not have. The Mac mini's runner takes the simulator lane. A push the PC did not build, or a PC that is offline, runs every job on GitHub's runners as before, which is the "box is down" switch. Each job mints one access token at its start and talks to Google over REST with it, instead of starting the cloud CLI for every read. Since 10/19/26 that order is the rule for every pipeline: the PC's runners first, then GitHub's, then Cloud Build. The deploys and the tests already follow it, and since the night of 10/19/26 the iPhone store builds do too: a store build alone runs on the PC's runner (one script, the same one Cloud Build runs), which starts it on Expo and waits for Expo and Apple at no cost; Cloud Build takes it when the PC is off, and also when the PC fails before Expo has the build, never after, so a fallback can't pay for a second Expo build. The app's web pipeline still starts on GitHub's runner, and moving it onto the same order is the next change.

8 · SMOKE TEST

Proof it serves, on dev before production both

Since 10/12/26 a function, the site and the API deploy their new revision with no traffic under a dev tag; the deploy script smoke-tests it at that URL and only then moves traffic, so a revision that fails never serves a player and production keeps what it had. Since 10/18/26 the revision stays there: every component is held at its dev rung after the smoke, and promotion is the owner's word, one Promote run that moves every held service to production in about 80 seconds on a PC runner, or a job image's climb through its twin. A push is a release to dev; production changes only when the owner says so, and since 10/19/26 the Promote run first calls the held revision three times at its dev address, so the first player after the promotion meets a running server instead of a cold start. The new version must be the ready one, and a live call must answer. A private function like the App Store watch is called with a signed token and a dry-run flag, so a deploy never sends a notice. When the desktop built the release, the desktop runs these same checks once the deploy is live, off the billed runner, and writes the verdict to the reports bucket; a push the desktop did not build is smoke-tested on the runner as before. The hourly watchdog alerts when a desktop built a release and never wrote its verdict. Since 10/13/26 that check runs from the desktop's main copy of the code, whichever folder pushed, and a check that crashes writes a failing verdict instead of none. Since 10/21/26 a release that deploys no backend skips the backend smoke at once instead of waiting for a deploy that never starts, and every release ends with a profile of its stages, runs and smoke times, set beside the last one's.

8½ · THE PHONE'S RUNG

An iPhone update runs on a simulator first phone updates

Since 10/13/26 an iPhone update goes from the desktop to Expo's dev channel, which only the Mac mini's iPhone simulator listens to. The simulator opens the app, waits only until the app's own update log says the download is in (about 10 to 20 s), and Maestro's cold start checks that the sign-in screen draws and that the version line shows this build's stamp: about 40 s in all, down from 71, with one more round allowed for a slow download. The verdict, with its download wait, Maestro's time and the rounds, goes to the reports bucket and the Release Dashboard's iPhone table. On the owner's "ship it" the push republishes that very update to members' iPhones; with no passing verdict it refuses. A new native version needs a store build and a new simulator copy, and a guard refuses any change that would cut installed phones off from updates.

9 · IF IT FAILS

Nothing breaks for players both

The old version keeps serving. GitHub files an issue with the failure in its title; Cloud Build files the same issue from inside the build, and its ERROR line emails the owner and posts to Slack one plain message that says what broke, what still worked and what to do. A Sentry alarm names the release, the failing line on GitHub and the suspect commit. If the preferred cloud is down, the other takes the push after 75 seconds; a result commit that races a session's push rebases and retries. Since 11/7/26 Terraform's apply tries once more when its first call to Google timed out before the state was read (a ship's twenty jobs on the PC can crowd one TLS handshake out; that day's rerun applied the same plan in 4 seconds), instead of failing the run and filing an alert for it.

How it got here

11/7/26

The Android update is published once, at the PC's build

A release's phone step sends the update both apps take. The iPhone's half was already the very update the PC published to the dev channel and the simulator passed, republished to production in about 9 s. The Android half was uploaded a second time from the same bundle, done at 21 s, so the step waited on it (the 11/6/26 10:24 AM PT release: 24 s of a 45 s App job). The PC's build now publishes Android's half to the dev channel too, beside the source maps' upload and before the bundle is packed, and writes its update group into the bundle. Every pipeline's publish reads it from there and republishes that group to production; a bundle without it, or a refused republish, uploads as before, so the Android app still gets its update. the phone step about 24 s → 12 s and the App job about 45 s → 33 s (expected; the next phone release measures it)

11/6/26

A quoting fault in a workflow step is refused at the commit

The 9:35 AM PT release published no web app and no iPhone update: the App job's step that links in the runner's kept packages died in 0 s on a bash syntax error (an apostrophe inside a ${BASE:+…} word, written that morning), every publish step after it was skipped, and the wait for the web publish failed at 300 s. A quoting fault is a parse error bash finds without running a thing, so a test now hands every bash block of every workflow, composite action and Cloud Build config (642 of them) to bash -n, and the PC's Linux gate and the Tests run refuse such a commit before it reaches main. The fix itself shipped 17 minutes later as a release fix, and the web and the phone went live at 9:53 AM PT. a broken App run caught at the commit, not by a release

11/6/26

Two cuts measured: the lock's delta, and the Tests Image's load

In the 2:35 AM PT release the App job installed its packages from scratch for 162 s because the lock file had moved, on a PC running the whole release at 100% CPU; the Tests Image was rebuilding beside it (307 s of docker build), since the requirements had changed. The install now adds only what the lock moved, over a copy of the newest kept install: 23 s in the 9:52 AM PT release, whose lock had moved too. And the 9:35 AM PT release, with no Tests Image build beside it, deployed every backend component in 34 s against 98 s, planned Terraform in 31 s against 93 s and smoke-tested the backend in 18.5 s against 47.5 s: the morning's slow release was the PC's load, not its jobs. install 162 s → 23 s on a moved lock; about 60 s off a release's deploy and Terraform when no image build runs beside it

11/1/26

The release captain's ship is one command

When Gene says "ship", the release captain ships the dock from the desktop's warm checkout, the copy of the code that is kept tested and built as each change lands. The captain had to look up how to run it there each time: on the 1:24 AM release that took 33 of the 47 seconds before the push, against 14 seconds for the ship itself. The captain's ship is now one named command, the same ship the board's SHIP card starts, so it begins in the captain's first step.

11/1/26

A release soon after a landing finds its code already on the dev rung

Each change that lands on the dock is warmed on the desktop: its tests, its images and its app build, so a release later only pushes. The warm then deployed the change to the dev rung, where a release promotes what is already built and smoke-tested instead of building it again. But the dev deploy came only once the warm had passed, and a release that followed within a couple of minutes stopped the warm first. In the 10/31/26 10:28 PM PT release, every service then built a new revision, and the deploy took 69 s against 39 s for the same services before. The dev deploy now starts as the warm starts: the change already passed its Linux tests to reach the dock, and the dev deploy waits for the warm's images by itself. The release's cancel of a dock deploy still running stays as it was, because waiting for one that has only just started would cost more than it saves. about 30 s of the release's deploy when it comes two minutes or more after a change that deploys nothing of its own (expected; the next such release measures it)

11/1/26

A smoke test waits out a busy service, and asks once more for a refused sign-in

Every deploy and every release ends with smoke tests: a request to each service that must come back right. Two kinds of failure were not the service's fault. Cloud Run answers 429 when it has no instance ready yet, and the 10/31/26 release counted that as the alert mailer failing. And once, Google refused the smoke's own sign-in token on GitHub's runner, which failed a deploy the next push passed 3 minutes later. A 429 is now waited out, as long as the service's own Retry-After says, with backoff, up to 60 s in all, before it counts. A refused token is asked for once more, 3 s later, before the deploy fails, and the failure says it tried twice. no false smoke failures from a cold service or a single token refusal

10/31/26

A release no longer waits for the dock's own deploy of the same code to step aside

Each push to the dock deploys its code to the dev rung, on the same desktop runners a release then needs. When the release followed within a few minutes, its jobs found those runners still busy. Each one had to wait until it queued, cancelled the dock's job for its own component and waited again while that runner wound down: in the 4dcac029 release, eight jobs waited 24 to 27 s that way. The ship now cancels the dock's running deploys itself, just before it pushes, so the runners are free when the release's jobs arrive. It never cancels while the dock's run is migrating the database: it waits up to 30 s for that step (6 s normally) and otherwise leaves the run alone, as before. A dock run cancelled this way, or by a newer dock push, no longer opens an alert issue. about 25 s of the release's deploy when it follows a dock push within minutes (expected; the next such release measures it)

10/31/26

A dropped connection to GitHub no longer holds a release for 16 to 20 seconds

A push to main starts about twenty jobs on the desktop's runners in the same second, and each downloads the commit from GitHub. In the 7:44 PM PT release, five of those connections were dropped at the handshake. GitHub's own checkout step waited 10 to 16 s before it tried again, which made those five the slowest jobs of the release: Deploy App Store Watch 35 s, of which checkout took 18 s, and Terraform Plan 41 s, of which checkout took 17 s. It had happened in 3 of the 8 releases before, at up to 20 s, while the network was fine. The desktop's jobs now check out with a small script of their own, kept on the runners' shared folder. It retries a dropped fetch after 1 s, and it does not fetch at all when the runner already holds the commit. GitHub's checkout takes over whenever the script cannot, and on GitHub's own runners. 10 to 20 s → about 1 to 3 s in the releases where a connection drops (expected; the next such release measures it)

10/31/26

The Release Dashboard's metrics job reads GitHub at its own pace again

That morning every GitHub read moved onto one shared, throttled client, which lets one process send 120 reads a minute. That suits a person's command, but the metrics job must read many hundreds of pages a run either way. Its runs failed at the 121st read until they learned to wait the minute out, and its first good run took 1,528 s against 228 to 439 s before. The throttle saved no quota, since the hour's last reads are kept for everyone else by a separate floor. The job's three collectors now read up to 600 a minute, under GitHub's own limit of 900. 1,528 s → about 5 minutes for a full metrics run (expected; the next run measures it)

10/31/26

A release of pipeline tools alone is skipped by the backend smoke at once

The 3:44 PM PT release deployed only the CI metrics job, one of the pipeline's own tools, which are never smoked after a release (their own deploy checks them on its dev rung). The backend smoke still waited 50.5 s for that job's deploy before writing "skipped". It already asks the push's own deploy decision, in about a second, to skip a push that deploys nothing; now the same answer skips a push that deploys only pipeline tools, and the verdict names them. 50.5 s → about 1 s of backend smoke on a pipeline-tools-only release (expected; the next one measures it)

10/31/26

The web smoke stops waiting for a build that is not coming

The 3:29 PM PT release's web publish failed on GitHub's runner, so no new web build could go live, and the desktop's web smoke waited 381 s until the fix's publish happened to serve the same app files; with no fix it would have waited out its whole 30-minute bound and then smoked a build it never saw. Every 30 s of that wait it now asks how the push's App run ended, and a run that finished without publishing ends the wait with a "not published" verdict and no smoke (the run's own failure alert covers the build). A running, green or unreadable run keeps the wait. 381 s → about 30 s of waiting after a failed publish (expected; the next failed publish measures it)

10/28/26

A blip in the runners' network no longer sends a deploy to GitHub twice

The 12:54 AM PT release went from Gene's word to its last run done in 109 s, with the Deploy wall at 40 s against 42 s before. But three component jobs (the vote webhook, the usage checks and the push decider) read their build mark during a one-minute blip in the runners' connections to GitHub and Google (38 failed calls on the network probe), took the failed read for "the desktop did not build this", and dispatched their deploys again on GitHub's billed runner: 54 s of extra runs and a second deploy each. A failed read is now asked again, four tries three seconds apart, and only a real "not found" means no mark. three re-dispatched deploys to none on a blip (expected; the next release with one measures it)

10/27/26

Measured: every component job started two seconds into the run, and a full redeploy found the runners' ceiling

The 1:18 AM PT release, the first on the cut below, redeployed all seventeen components (a shared-library pin moved), and every component job started 2 s after the run began, against 14 to 15 s on the two releases before. The Deploy wall still grew, 45 s against 38 s, because seventeen deploys ran instead of a few and the API's job ran its boot check and migration (11 s) and deploy (10 s) behind 12 s of small steps; and with twenty-two jobs at once on twenty runners the site's deploy waited 21 s for a free one. Two cuts follow: twenty-four runners (the box had 27 GB free; built the same night), and a component job that trusts the desktop's answer instead of walking the import graph a second time, 2 s off every touched component. component start 2 s against 14 to 15 s (measured); runner wait 21 s to 0 on a full redeploy and 2 s per touched component (expected; the next release measures them)

10/27/26

The serial "What this push deploys" job leaves the critical path

The two 10/26 profiles named the same next cut: on every release the first 14 to 15 s of the Deploy run was one job deciding which components the push touches, picking the runner and writing the settings snapshot, then handing off to the next runner, while the desktop's build mark already carried that decision. Now, on a desktop push to main, that job is skipped: every component job starts the moment the run does, on one of the twenty desktop runners, and its first step reads the mark itself (the mark's answer for that component, or the same import walk when the mark does not answer this push); a job the push does not touch ends in a few seconds. A push the desktop did not build finds no mark, skips, and dispatches its own deploy on GitHub's runner, the runner watchdog's shape. The snapshot Cloud Build renders from is written by a job beside the components that nobody waits on. A push to the waiting batch, a hand dispatch and GitHub's runner keep the deciding job. about 12 to 14 s off every release's Deploy (expected; the next release measures it)

10/25/26

A release in 58 seconds, after one that started in the wrong place

The afternoon's first release took 3 m 36 s from the owner's word to its last run, and 2 m 53 s of that came before the push. The ship had been started from a copy of the code that was already live, so it spent 51 s on an app build and 70 s on images before the check that guards installed phones stopped it, and it had to start again from the copy that holds the waiting batch. The ship now refuses such a copy before its first stage and names the right one. The same release's app build took 21 s on an app that had not changed: the batch's background run skips the app when it matches what is live, so it never kept the guard's answer, and the ship worked it out cold. The background run now keeps that answer for the ship. Its images took 18 s to find six that were already built, because the mistaken build had replaced the desktop's memo of them; the memo now keeps each image's last four builds, so a build in between no longer sends the ship to the registry. The tests that pick only the checks a change can break also missed one that reads whole folders, and a copy of the payment handle that it refuses reached main unseen; a check that lists folders now runs on any such change. The next release took 58 s from the word to its last run: 16 s before the push, 6 s to push and 31 s in the cloud. word to last run 3 m 36 s to 58 s (measured, a smaller release); app build on an unchanged app 21 s to 3 s, images already built 18 s to 4 s (measured); a build in between costs no registry lookups (expected; the next release measures it)

10/24/26

Deploys that promote what dev already runs, and runners that keep their own actions

The morning's web and iPhone release took 2 m 04 s from the owner's word to the last run. Its deploy ran 42 s against 60 s the phone release before, and its slowest deploy job 28 s against 47 s, because an iPhone ship no longer adds an empty commit of its own: the release's commit is the one already deployed on dev, so every service switched production to that revision instead of building again, and the iPhone update now follows the web publish by itself. The release's longest stage was Terraform's plan job, 47 s, of which 30 s passed before its first step: the runner downloads every action a job uses from GitHub, the download of one failed, and the runner waited 23 s before trying again. The desktop's twenty runners now share one copy of every action the workflows use, refreshed each morning, so a job reads them from the box instead of downloading them. deploy 60 s to 42 s, slowest deploy job 47 s to 28 s (measured); up to 30 s off a job's setup when a download fails, about 1 s on every job otherwise (expected; the next release measures it)

10/23/26

Every deploy job under 30 seconds: the side work moves onto the CI bus

The evening release carried four cuts, and every service's deploy job finished in 21 to 27 s against 35 to 51 s the release before. Copying the shared pickleball code beside each function, instead of having pip build a package of plain files, took the install from 3.9 s to under one. Telling Sentry a release is live became one message on the CI bus, under a second, and a listener in the cloud marks the release once for the whole push. App Store Watch's deploy check became a ping instead of a 13.5 s read of Apple; the listener now reads Apple with the new key after the production deploy, and a failure becomes an alert. Terraform applied the plan the waiting batch had made, so its plan job dropped from 48 s to 33 s; this release's apply was long, 6 minutes, because it created a Firestore expiry rule, which Google applies slowly and only once. The desktop's release smoke now finishes in 16 s against 58 s. deploy 51 s to 45 s, slowest deploy job 35 s to 27 s, release smoke 58 s to 16 s (measured)

10/23/26

The release that was already on dev, and Terraform planned ahead

The afternoon's twelve changes were the first release after the waiting batch started deploying itself to the dev rung as it filled: by the time the owner said ship, every service already ran its new revision on dev, smoke-tested there. The ship built nothing and only switched production over to those exact revisions, so its deploy took 51 s against 68 s the release before, and its slowest job 35 s against 54 s. The morning's read cuts measured out too: no function deploy asked Cloud Run the same thing twice. What still padded each switch-over was 7 s of reads and records, one after another, ahead of a 2.4 s traffic change; those now run side by side. That release's longest stage was Terraform, 78 s, because it carried a new schedule: Terraform now plans while the batch waits, and the ship applies that plan instead of planning again. deploy 68 s to 51 s (measured); about 3 s more off each switch-over and about 30 s off a ship that changes infrastructure (expected; the next release measures it)

10/23/26

A nine-change release in 83 seconds, and the reads that padded every deploy

The morning's release carried the 8 AM run's switch fix and eight court-booking changes: 1 m 23 s from the owner's word to the last run done, its deploy 63 s against 66 s the release before, its smoke 15.3 s. The slowest job was the daily image's, 49 s: of its 38 s step, deploying the four jobs took 7 s and Google starting each job's twin to prove the new image took about 23 s, which no setting here shortens. The functions' deploys, 40 to 42 s, sat right behind it, and each spent about 9 s asking Cloud Run the same things one at a time: which revision serves, which one is new, where its dev address is, and where production lives. Each function now reads its service once before the deploy and once after, the smoke's three checks ask at the same time, and a job's twin starts before the read that keeps its rollback. about 5 s off each function's deploy and 2 s off each job's (expected; the next release measures it)

10/22/26

A stuck desktop runner is restarted, then bypassed, and the infrastructure step keeps its downloads

That morning three deploy jobs sat waiting while every desktop runner showed idle, until someone restarted them by hand. Now the cloud watches each job meant for the desktop. If one waits two minutes, it posts a message on a channel of its own, kept apart from the one players' actions travel on, and the desktop restarts only the runners that are not mid-job. If the job still waits three minutes later, it moves to GitHub's own runner. A job that started but whose end GitHub never recorded is flagged rather than retried, since no restart helps it. The release that brought this took 8 minutes from the owner's word to its last run, against 2 m 12 s for the backend release before it, which changed no infrastructure; its deploy took 95 s against 110. Almost all of the difference was one step: creating the rule that clears old job records took 5 m 53 s, a cost Google charges once when such a rule is made, never again. The infrastructure step's ordinary time is about 88 s, and about 30 s of that is downloading the same providers twice, once in each of its two jobs. The desktop's runners now keep those downloads in a folder they all share. infrastructure step about 88 s → 60 s (expected); its 5 m 53 s rule creation one-time

10/22/26

The standby pipeline catches up with the main one, and one of its gaps was a real fault

Every release goes out on the main pipeline unless it is down, when a standby one in Google's cloud takes over. A week of speedups had gone into the main pipeline alone, so all of that week's changes were checked against the standby. Most were already in both, because they live in the scripts both run. Four were not. One was a fault: on the standby, the API skipped the hold that keeps every new version at its test stage until the owner says go, and would have gone straight to production. It now holds like everything else. The other three were speedups: each deployment record read its access key from storage again, the API's settings checks did the same, and the tests reran a tree the desktop had already tested. The standby now reads the key once per deploy, makes one settings key, and skips a tested tree. A test now holds the two pipelines to the same four, and every main-pipeline change brings its standby twin in the same change. The main pipeline's records also stopped reading the key at all: each deploy is handed its own by GitHub, and the records start beside the deploy instead of before it. standby: API hold fixed; records' key read once per deploy instead of once per record (expected)

10/22/26

The first release off a warm queue: 107 s from the owner's word to the last run done

Ready changes now wait on a queue, and each one added there runs the release's own checks in the background (the tests, the images, the app build and the Simulator), so by the time the owner says ship they have all passed. The first release this way, a web and iPhone release that also put one server job live, took 107 s from the owner's word to its last run: 29 s for the steps before the push, 10 s to push, 68 s for the cloud. Against the iPhone release before it: tests 21 s → 0, app build 72 s → 9, Simulator 40 s → 0, before the push about 122 s → 29. The site's pages took 33 s instead of 94, because the push published only the one page it changed. The cloud's longest run was the job's deploy, 40 s: its test copy deployed and ran first, then production. About 9.5 s of those 40 were the three GitHub records of the release, each fetching its token again before starting, and the first one finishing before the job's deploy could begin. The token is now fetched once per deploy, and the first record runs while the deploy does. before the push about 122 s → 29 s (measured); site pages 94 s → 33 s (measured); a job's deploy about 40 s → 33 s (expected)

10/22/26

An app-only iPhone release: 151 s from first stage to production, against 227 s the release before

The next iPhone release changed only the app, so no server deployed. Against the one before: the stages before the push took about 122 s instead of 151, and first stage to the end 151 s instead of 227. The app build took 72 s instead of 100, because it reused the export it had already built and tested. The "What's new" notes, a 62 s call to Claude, were written while the build ran instead of after the publish. The tests took 21 s instead of 60: fewer suites applied, the slowest started first, and none was running beside a heavy build. Part of the saving is this release being lighter, so the like-for-like gains are the reused build and the notes off the path. What is left is the app build and the Simulator's 40 s check. The release's last run was the site's pages, 94 s instead of 30: one Build Pipeline entry republished all eight pages, and one upload stalled. A push now publishes only the pages whose inputs it changed (on this release, one page instead of eight); a change to the shared page code still publishes all. first stage to production 227 s → 151 s (measured, an app-only release); site pages 8 → 1 for a one-page change (expected)

10/22/26

Measured: the slowest function's deploy is down to 36 s, and the night's releases run clean

The next release carried the fixes for the night's two side-by-side mistakes, a permission race and a test that expected a fixed order, and ran clean: every check, the build beside the tests, the deploy with each service and job promoted inside it, and the branch the next release starts from moved forward with main by itself. The function that started the night at 84 s deployed in 36 s. Before the push, the stages now take about 151 s on an iPhone release (the app build's 100 s with the tests and images beside it, then the Simulator's 40 s and the push's 11 s), and the deploy after it 69 s. What is left is the app build itself and the app's own start in the Simulator. slowest function deploy 84 s → 36 s (measured)

10/22/26

An iPhone release builds its app while its tests run, about 45 s sooner to the push

The owner dropped the old rule that a failed test builds nothing: it dates from when a build cost money, and on the desktop it is free. So an iPhone release now builds its app at the same time as the Python tests, and only what leaves the desktop (the publish, the push, a store upload) waits for the tests to pass. The stages before the push went from about 189 s in a row (tests, then the build, then the Simulator) to about 144 s: the build's 104 s with the tests and the images beside it, then the Simulator's 40 s. The tests take longer sharing the machine, 66 s, but nothing waits for them now. The same release found two of the night's own cuts too eager: two permission grants on one service, run at the same time, collided on the service's permission record and stopped that service at its test step, its users kept on the version before; it now checks what is already granted and adds only what is missing, one at a time. The remaining variable in a deploy is the cloud's own switch of traffic to the new version, from 7 to 37 s. about 189 s → 144 s before an iPhone release's push (measured)

10/22/26

Measured: a function's deploy fell from 84 s to 47 s, and the tests now start their slowest suites first

The next release measured last night's deploy cuts. The slowest function's deploy step fell from 84 s to 47 s, more than expected: with fewer traffic changes Cloud Run moved production in 7 s instead of 37. The API's step fell from 67 s to 36 s, and the whole deploy from 114 s to 83 s, with every service and job promoted inside it and all 13 smoke tests passing on production. What is left in a function's deploy is the deploy itself, its test against the new version and the switch, each a few seconds. The longest stage before a release's push is now its tests, 45 s. Every core was already busy, but the suites started in file order, so a slow one could start last and run alone; they now start slowest first, by the last run's times, which replayed on the same machine gives 31 s instead of 42.5. Two smaller cuts went in beside it. The two functions that grant permissions on every deploy now grant them side by side instead of one after another, 4 s and 8 s of their deploys. And the smoke run after a release no longer runs each job again when production already points at the exact image its test copy ran and passed minutes before: one lookup shows it, and anything uncertain still runs. On the live jobs that took the daily image's four from 82 s to 17 s. deploy 114 s → 83 s (measured); tests about 45 s → 34 s (expected); job smoke 82 s → 17 s (measured)

10/21/26

The first release promoted in its own deploy, and a function's deploy makes one traffic change fewer

The first release to climb straight to production went live 1 m 54 s after its push, with every service and job promoted inside its own deploy and no separate promotion. A second release had to fix it first: an unquoted colon in the deploy workflow made GitHub refuse the file, so nothing deployed for two minutes, and a new check now reads every workflow the way GitHub does. Its slowest deploy, a function at 84 s, spent 54 s on three changes to its traffic: hold production, tag the new version for its test, then promote. A function now deploys untrafficked and tagged in the same call, as the API already did, which drops two of the three. The release's record of each step, written to GitHub, no longer holds up the promotion either: about 4 s that now runs while Cloud Run moves the traffic. What is left is Cloud Run's own wait to move the traffic, 36 s for the API, which the deploy waits out so it never says "promoted" before production serves the new version. about 21 s off each function's deploy, 4 s off the API's (expected)

10/21/26

A promoted release reaches production in its own deploy, and its images build beside the tests

Every service has always gone to its dev address first and been smoke-tested there; since the hold of 10/18, production waited for a separate promotion afterwards. A release the owner promotes now says so on its last commit, and its own deploy takes each service from the dev test straight to production in the same job; the separate promotion runs only for something that fell back to Cloud Build and stopped at dev. The promotion after the deploy was 52 s of a 5 m 58 s release. Before the push, the images now build at the same time as the tests instead of after them, up to 30 s on a release with backend changes. The next release measures both. 52 s promotion → 0 s on a promoted release (expected)

10/21/26

A promoted release is smoke-tested whole, and its images build in one wave

A release promoted to production runs more than one Deploy run on its commit: the push's own, then one per job climbing to production. The backend smoke read only the newest of them, so one release's API and five functions went untested until a hand run passed all nine. It now reads every Deploy run of the commit. The profile counts only the release's own runs, so a later manual run on the same commit no longer stretches its time. The images, about nine built four at a time in 122 s, now build all at once, up to twelve: on a phone release they run beside the app build and the Simulator and wait for nobody, but on a web or backend release they are the longest stage before the push; the next one measures it.

10/21/26

Every release is profiled, and the smoke no longer waits for a deploy that isn't coming

Each release now ends with a profile, written once its smoke tests pass: the stages before the push, every workflow run's jobs with how long each waited for a runner and its slowest steps, and the backend and web smoke times. A session compares it with the last release of the same kind and turns what it finds into the next cut. The first one found that a release changing no backend code still waited three minutes for a Deploy run before writing a "skipped" verdict. The smoke now asks the same deploy-paths check the push uses, which already knows in about a second that nothing deploys, and skips the wait. 194 s → 0 s backend smoke on a release with no backend change

10/19/26

Apple tells us when a review moves

The App Store watch read Apple every 30 minutes, but only while it was "armed", and only the pipeline's own submissions armed it: a build submitted by hand in App Store Connect, as both of that night's reviews were, went unseen until the next daily read. Apple's App Store Connect now sends a signed webhook for every change to the App Store version, a TestFlight build's beta review and a build's upload; the API checks the signature and wakes the watch, which reads Apple and sends the email and Slack note within seconds. Nothing arms anything any more, and the daily read stays as the backstop. every review change seen, however it was submitted

10/19/26

Store builds from the PC, on a bigger Expo machine

Profiled end to end, store build 18 took 16 minutes from start to a build Apple accepted, and 7 min 35 s of it was Xcode compiling on Expo's standard machine with nothing cached; the App Store submission then took another 36 minutes because two missing items (a price and the 13-inch iPad screenshots) only showed up at the final "Add for Review". Now: the production build asks Expo for its large machine and keeps a compiler cache between builds; every store build lists what App Store Connect still lacks while Expo is building, so it gets fixed in parallel; a store build alone starts from the PC's own runner, where waiting on Expo and Apple costs nothing, with Cloud Build standing by; and both run the same store-build script, so the two paths can't drift. The large machine's minutes come out of the Expo plan's build credit, measured on the next build. blockers found while Expo builds, not after

10/19/26

A promotion warms what it promotes

The first match after an evening promotion of the API took 7.4 seconds on the server: the dev rung's smoke had started an instance of the new revision 18 minutes earlier, and it had scaled to zero. Promote now calls the held revision three times at its dev address just before it moves the traffic, so an instance of exactly that revision is running when the first player arrives; a warm-up that fails says so and the promotion goes ahead. 7.4 s → about 1 s for the first tap after a promotion

10/19/26

Every push is gated, not only the app's

A backend push rebased by hand after its marks had been made reached the runners with no PC mark, and its deploys ran on GitHub's billed runners. The push gate now runs the PC's tests and image build for every push to main, a few seconds when the marks already answer, so no push leaves the PC unmarked. The rule is written for every project, and the pipeline enforces it. no unmarked push to main

10/19/26

The PC, then GitHub, then Cloud Build

The order of preference is now a rule, for cost and speed: the PC's own runners cost nothing, GitHub's start faster than Cloud Build, and Cloud Build stands by. The deploys and the tests follow it, and that night the iPhone store builds did too (below); the app's web pipeline, which still starts on GitHub's runner, is the next to move.

10/19/26

Store builds without the extra work

An iPhone store build had re-run the PC's tests and re-shipped the web app. Every iOS store build now starts from Cloud Build, reads the PC's tested mark instead of testing again, and skips the web steps when only the store build ships; its config files (Firebase's among them) are checked on the PC before the push and first thing in Cloud Build, so a missing one fails in seconds, not twenty minutes in. A finished build can go to Apple through Expo's own submit or from the Mac mini with Apple's Transporter, and each platform carries its own runtime version, so an Android change never cuts the iPhones off from updates. no second test run per store build

10/19/26

Terraform's plan survives a slow read and its own lock

One plan spent 76 seconds reading a Sentry setting and hit its two-minute limit; the killed plan left its state lock behind, and the next push's plan failed in one second on it. The plan now has five minutes, plan and apply wait up to two minutes for a lock instead of failing, a plan that fails or is cancelled releases its own lock, and the format check gates the plan instead of only printing a diff. a killed plan no longer fails the next push

10/19/26

The web smoke for every web publish

A push that changed only the app's Cloud Build config republished the web app without its smoke test, because the push started the smoke only for changes inside the app's folder. It now starts for any push the app's pipeline publishes on, and a test holds the two lists equal. no web release unsmoked

10/18/26

The gate survives a main that moved

Four pushes in one evening lost a race: main moved while the gate was testing, and the push was refused after the work was done. The gate now fetches main again at its end and rebases onto it in place; a check whose inputs the rebase did not touch carries its mark to the new commit, one whose inputs it touched runs again there, and the push is asked once more and clears in seconds. The same evening the gate became one shared program, used by the owner's other projects with their own checks. a lost race costs seconds, not a rerun

10/18/26

Every graph within the hour

The Release Dashboard's deploy and test graphs had moved once a day, so a busy day's pushes showed up the next morning. The CI metrics job now runs every hour and republishes the page itself, reading only runs it has not seen. daily → hourly

10/18/26

A tested tree's Tests run bills nothing

For a tree the PC had already tested, the Tests job still started GitHub's runner to claim the push and find the mark, 17 seconds billed as a minute. The claim and the mark check are now a job of their own on the PC's runners, and the suites job is skipped outright, which GitHub never bills. The PC down, the claim job waits, GitHub never claims, and the standby runs the suites on Cloud Build. one billed minute less per push

10/18/26

Every function deploys as a prebuilt image

Cloud Functions' own deploy had rebuilt each function's container on Cloud Build on every deploy, about a minute and a billed build each, ten times a full push. Now each function's image is built once on the PC with the Functions Framework inside, and deployed by digest onto the function's own Cloud Run service through the same ladder; its address, its Pub/Sub trigger and its permissions stay. The first full run: every function in 24 to 46 seconds, no Cloud Build anywhere in it. 75–103 s → 24–46 s a function

10/18/26

The build gate answers from a memo

Before each push the PC asked the registry and the bucket about every image and zip to learn that nothing had changed, five minutes of asking. It now keeps a memo of what it last found or built, keyed by the git ids of each artifact's inputs, and asks only about what changed. 5 min → 1 min 23 s with nothing to build

10/18/26

Every backend component held at its dev rung

Production had taken every green push. Now a deploy ends with the new revision serving at its dev tag, smoke-tested there, and nothing moves until the owner says promote: one Promote run moves every held service at once (eleven services in 83 s on a PC runner), a job image climbs through its twin. The bus's events follow the vote writer's dev tag while the hold is on. a push is a release to dev; production is a word

10/18/26

The push itself tests and builds what it pushes

Two pushes had gone out with no PC mark, so every job ran on GitHub's billed runner and three job images were built and smoked there. Git's own pre-push hook now runs the PC's tests, the app build check and the image build for a push that changes the app, and refuses the push on a failure; a backend push goes straight out and says in a line which mark it lacks. And before any of that, a push whose head is not on top of main is refused in a second, so a build made on a stale base never reaches the runner. no unmarked app push; no stale-base push

10/18/26

One token per job, one snapshot per push

Each deploy job had started the cloud CLI four or five times to read a secret, write the settings snapshot, claim the push and tell Sentry. Each job now mints one access token at its start and talks to Google over REST, and the Deploy run's first job writes the settings snapshot once for every component. about 15 s off every deploy job

10/18/26

Promote all, as one job

Promoting the sixteen components had been sixteen dispatches. One Promote run with component=all moves every held service in one job on a PC runner, and reads each service's held revision itself. sixteen dispatches → one run

10/18/26

The pool stops asking Neon before every request

The API had pinged its database before each request's first statement, about 60 ms of every answer. The pool now recycles its connections under Neon's five-minute suspend instead, so an answer from the dev rung that took 650 ms takes about 370. Not a pipeline change, but measured on the same afternoon through the dev rung, which is what the rung is for.

10/15/26

A push claimed for GitHub the moment it starts

Jobs waiting their turn for a desktop runner could sit past the push decider's 75 seconds, and Cloud Build took the push over. Now, when the desktop's runners are seen online, the Deploy run's first job claims every component for GitHub at once, and each job runs whenever a runner frees up. The first push claimed all sixteen components and Cloud Build started nothing. 16 of 16 on the desktop, 0 to Cloud Build

10/15/26

A Cloud Build takeover really deploys

When Cloud Build did take a push, its decide step asked git whether the push touched the component, but its image has no git, so it answered no and deployed nothing while holding the claim; GitHub saw the claim and skipped too. A build the push decider starts now deploys without asking again, since the decider already found the push touches it.

10/15/26

Deploys run on the PC's own runners, four at a time

A push the PC built now deploys on GitHub Actions runners that run on the PC itself, in containers, so those jobs use no billed minutes. One runner took a push's sixteen jobs one at a time; four take them side by side. A push the PC did not build, or a PC that is offline, deploys on GitHub's runners as before. no billed minutes for a desktop-built push

10/15/26

The API always deploys where Docker is

The API's deploy starts its new image to check it boots and runs the database migration inside it, which needs Docker; the PC's runner containers have none. The API's job now always runs on GitHub's runner, and every other component stays on the PC.

10/14/26

Docker down hands the work to the cloud

Docker Desktop's own updates restarted it in the middle of a build. The PC now starts Docker and waits up to two minutes; if it is still down, the build and the tests say they were handed off and the cloud builds and tests that push instead of the release stopping.

10/14/26

Every web release smoke-tested by the push itself

Two web releases went live without their smoke test because the step depended on someone remembering to start it. The push now starts it, for every push that changes the app. no release goes unsmoked

10/14/26

A dev build tests only what its change can break

An app build for the dev rung now runs only the test files related to the files that changed; a build that ships runs every one of them, so nothing reaches members on a partial run.

10/14/26

The simulator test runs on the Mac mini itself

The iPhone simulator check now has its own workflow on the Mac mini's runner, so it no longer needs a connection from the PC to start.

10/13/26

One Deploy workflow instead of sixteen

Every backend component had its own deploy workflow. Now one Deploy run per push has a job per component, and its first job decides from the import walk which components the push touches, so the release reads as one run. The bot's commands that used to start GitHub workflows run on Cloud Run jobs instead. one run per push

10/13/26

Every iPhone update timed, stage by stage

The Release Dashboard's iPhone table now has a column per phone update: the export on the PC, the upload, the publish to the dev channel, and the Mac mini's simulator test with its download wait, Maestro's time and how many rounds it took. The first one showed the slowest stage plainly: a simulator just back from a restart took two rounds. every stage measured, every release

10/13/26

The simulator check in about 40 seconds

The simulator had waited a fixed 30 seconds for each update and opened the app twice. Now it watches the app's own update log, which says the download is in after a few seconds, and Maestro keeps its driver between runs. 71 s → about 40 s

10/13/26

The PC's app build side by side

Profiled first: the build is limited by the processor alone, never disk or memory, and the minifier is most of it. So the typecheck and tests, the two exports and the fingerprint check now run at the same time, and the source maps go to Sentry while the build uploads. 72 s → 56 s

10/13/26

Fewer billed GitHub minutes

The month had passed GitHub's included minutes. Each deploy's path filter is now generated from the import walk, so a push that touches no deploy's code starts no deploy; the tests on GitHub skip a commit the PC already tested in the same Linux image Cloud Build uses; and the workflow-file checks moved into the tests job. fewer runs per push

10/13/26

One deploy email per release

Every function's deploy had told Sentry it went live, so one push sent ten "deployed" emails. Now the first deploy of a push claims the release in the reports bucket and the rest skip it.

10/13/26

A smoke check that survives a deleted folder

A push from a temporary copy of the code left no smoke verdict: the check ran from that copy, which was deleted while it waited for the deploy. It now runs from the PC's main copy, and a check that crashes writes a failing verdict instead of none.

10/13/26

iPhone updates climb a ladder too

Every phone update had gone from the desktop's tests straight to members' phones. Now it is published first to Expo's dev channel, run on the Mac mini's iPhone simulator (a 2018 Intel Mac, so its copy of the app is built once by Expo's cloud and each update rides on it), checked by Maestro, and only the update that passed is republished to production. The first pass took 71 seconds. no untested update reaches a phone

10/13/26

The web app's first download a third smaller

Firestore's live updates and PostHog now load after the page draws, in files of their own, and the web export drops code nothing calls. The sign-up page, the first page new members see, shows its form 3 seconds sooner on a slow phone. 1,070 → 686 KB per first visit

10/12/26

A guard for the phones' native version

A store build that changed the app's native fingerprint had cut installed phones off from updates. The desktop build now compares the fingerprint with the installed store build's and refuses a change unless the owner approves it, with a store build to match.

10/11/26

Every pipeline on its own Cloud Build trigger, and the switch on

Seventeen triggers generated from the workflows' own path filters, one build file for the nine functions, the job images and the test image with theirs, the claim in every runner job. Each kind proved by hand first; GCB_TRIGGERS went on at 11:48 AM PT and the next pushes were claimed by GitHub while Cloud Build stepped aside in under a second. a GitHub outage stops nothing

10/11/26

One snapshot of settings for both paths

Every deploy job writes GitHub's Variables into one Secret Manager snapshot; a Cloud Build build renders its workflow's settings from it. The two paths can no longer deploy different values.

10/11/26

The desktop builds the images and sources

Docker on the desktop builds every job image and function zip whose content changed, before the push, from the committed tree; the deploys find them and skip their build. Images ship only what they run: the court booking image 1,086 → 347 MB, the daily 1,087 → 399 MB, the nightly 234 → 120 MB; the registry keeps ten versions. no build minutes on a push

10/11/26

The desktop smoke-tests the release

A deploy that finds the desktop's mark skips its smoke step; the desktop runs the same checks once the release is live and writes the verdict to the reports bucket. A missing verdict is an alert from the hourly watchdog. The web app the same way: tested and smoke-tested once, on the desktop, so a push costs about 40 runner seconds.

10/11/26

Sizes and Sentry on every release

Every deploy log prints the build's compressed size beside the last one's, and every deploy marks its release in Sentry with its commit, so an error names the release, the failing line on GitHub and the suspect commit. Sentry alarms post to Slack with the actions of Sentry's own app.

10/11/26

A standby that costs seconds

A Cloud Build build that steps aside was billing about 83 s of image pulls. Every Python step moved to the slim image, and then the claim became a gate build of its own: one small step that claims first and starts the real build only when Cloud Build wins. a step-aside costs seconds

10/12/26

A standby that costs nothing

GitHub rounds every job up to a whole minute; Cloud Build bills by the second. The gate builds were cheap, but at 40 to 60 pushes a day they were still about 115 build-minutes of finding out that GitHub had already claimed. The push decider replaced them: a small function on GitHub's push webhook notes which pipelines a push touches, and a one-minute sweep starts Cloud Build only for a push GitHub never claimed. no Cloud Build run until it is needed

10/12/26

The app climbs a ladder

Three releases had reached production while dev still served an older build. Now the desktop build is promoted to dev before anything is pushed, and both pipelines refuse a build dev is not serving: a rung above never runs a newer build than the rung below. dev is a real rung, enforced

10/12/26

The backend climbs too

The functions, the site and the API had no rung below production: a bad build served players until the smoke caught it. Now the deploy script pins production, deploys the new revision with no traffic, smoke-tests it at its dev URL, and only then moves traffic. A revision that fails on dev never serves a request. no bad build reaches a player

10/11/26

Result commits survive a race

Three workflows that commit their result to the repo could not rebase onto a session's push (no git identity for the rebase; the first time in 51 runs that main had moved under one). Fixed in all three, and a failure now names the git error, never the sync's own success line.

10/9/26

Test selector fixed and fast

A pipeline script change no longer runs all 8,000+ checks, and the selector parses each file once. 75 s → 8 s per tests build

10/9/26

Timeouts sized to real runs

41 workflow timeouts set to 150% of the average successful run. A failed or timed-out /pb command now replies in its chat with the error.

10/9/26

The API built from its own Dockerfile

The same image on both paths, deployed as an image, migration first. First Docker-built version live at 5:04 PM PT.

10/9/26

A GitHub path for every pipeline

Six functions, the site, five scheduled-job deploys, the test image, the API and the tests, with the same deploy scripts on both paths. The app's GitHub path is Black Widow's. about 2 min faster a run

10/9/26

GitHub by default

The owner chose GitHub that afternoon for speed; the switch went on that evening after a proving run of the API, the tests and the app. The decider now prefers GitHub unless it's blocked.

10/9/26

Cloud Build on its own trigger

The owner's console steps connected Cloud Build to the repo; Terraform linked it and created triggers for the tests and the API, switched off, plus the per-push claim that lets the two tracks share a push.

10/9/26

One scheduler job per task

The Neon usage and API traffic checks left the nightly job for their own jobs and schedules, and the App Store review watch got its own function, every 30 minutes, signed calls only.

Numbers from ci-pipelines.json, measured after each production release by the CI metrics job, last Mon 11/16/26 4:32 AM PT, release-profiles.json for each machine's stages, recorded after every release, and app-build-benchmark.json for the two paths. The diagrams and steps are from Snow White's build-pipelines report; the app's ladder is Black Widow's.