CI/CD Optimizations, Part 2: Release Optimization Electric Boogaloo

Previously: CI/CD Optimizations, Part 1: PR Checks

I turned to releases because we were cutting a lot of them. Eight in 24 hours that week, each one a 13 to 30 minute wait, and I wanted faster iterations into production while I was prepping for launch. I even went and optimized Pilum while I was at it, with language-specific build optimizations and pre-caching across runners. That’s its own article.

Part 1 got pull requests down to about 2.5 minutes. Releases are a different pipeline with a different bottleneck. A Statio desktop release builds a Tauri app on four platforms in parallel (Windows, Ubuntu 22.04, macOS on Apple Silicon and macOS on Intel), then publishes the installers, the mcphub package and the Homebrew cask. When things went right it took about 13 minutes. When they didn’t, it took 25 or 30, and I couldn’t tell in advance which one I’d get.

Here are all eight releases from that stretch:

Two bars are about twice as tall as the rest, and they have different causes. That’s this whole post.

Where releases stood

Releases wait on the slowest of the four build legs. Most of the time that’s Windows, at 8.5 to 10.5 minutes, against 3 to 4 minutes on macOS. The build itself is a Rust compile, not a JavaScript one. On the Windows leg “Build the app” took 235 to 292 seconds on a normal run.

The cache that ate the other caches (#181)

The run-keyed Go cache from #170 (see Part 1) writes a new entry on every run, about 0.56 GiB each, PRs included. The repo’s cache usage hit 12.5 GiB against GitHub’s 10 GiB limit, so least-recently-used eviction ran constantly.

The caches used least are the Rust ones for the Tauri desktop build, which only run on a release. So they were always the oldest thing in the store and always the first to go. The third release on Oct 7 took 30 minutes. Its Ubuntu leg spent 738 seconds building, against about 230 normally, and its Windows leg spent 818. The Rust cache “restore” took 1 to 3 seconds. A real restore of a Rust target/ directory takes 15 to 35 here. It was a miss, and it looked like a Tauri problem until I listed the cache entries.

$ gh api repos/.../actions/cache/usage
active caches size: 12.5 GiB   count: 45

0.57 GiB  go-Linux-go-a6a22cc...      x3 near-duplicates
0.56 GiB  go-Linux-lint-a6a22cc...    x4 near-duplicates

A miss costs twice. The Windows leg on that run also spent 216 seconds in “Post Rust cache” and 178 in “Post Setup Go”, saving fresh caches to replace the ones that had been evicted. That’s six and a half minutes of uploads on top of a slow build, and they’d be evicted again before the next release.

The fix (restore on every run, save Go caches only on push to main) is covered in Part 1. It cut Go cache writes 5 to 6 times.

The setup-go cache went the same way

The desktop workflow also used setup-go’s built-in cache, about 0.6 to 0.7 GiB per leg. On the 30-minute run it logged Cache is not found on every leg. The mcphub sidecar build took 54 seconds on Ubuntu and 86 on Windows, against 2 to 3 and 16 to 27 when the cache hit. On the next release it hit again on all four legs. Nothing was wrong with it. It was evicted along with everything else.

#181 turned it off anyway, as part of cutting cache writes. Four legs each writing over half a gigabyte is a big share of a 10 GiB budget. That had a cost I didn’t see until later. Without it the sidecar build took 36 to 55 seconds on Ubuntu and 70 to 75 on Windows, every release. #184 brought back a smaller cache aimed at the sidecar.

What #181 did not change

• No Cargo profile change. There’s no [profile.release] in the tree, so Cargo’s defaults apply (codegen-units=16, lto=false), already the compile-fast setting. Tauri templates often ship lto=true, which would make this worse.
• rust-cache stays. It was configured correctly; it was being evicted, not misconfigured.

The first release after the fix took 25 minutes

The release right after #181 merged took 25 minutes, and for a few minutes I thought the fix had made things worse. It hadn’t. The Rust cache hit fine. The Ubuntu leg spent 862 seconds in “Install dependencies”, which is apt-get update plus downloading four packages Tauri needs on Linux. That step is normally 30 to 50 seconds. A slow mirror, nothing else.

Why I didn’t move to ubuntu-24.04 (#183)

My first instinct was that ubuntu-22.04 was a stale pin and 24.04 would have faster mirrors. That would have shipped a bug to users.

Tauri’s AppImage docs say to build on the oldest base system that still has WebKitGTK 4.1, because the build host sets the glibc floor for the bundle you ship. Ubuntu 22.04 is glibc 2.35. 24.04 is 2.39. A Tauri 2 project that tried it found its 24.04-built AppImage, .deb and .rpm wouldn’t start on 22.04, and Debian 12 (glibc 2.36) breaks the same way. tauri-action’s 24.04 request was closed as not planned, and a Tauri maintainer confirms the rule.

CI would never have caught that. Users would have, by telling me the app won’t launch.

I almost merged it. Before I did, I went back to find out why we’d pinned 22.04 in the first place, and it’s this: a known Tauri issue with newer Ubuntu, with the fix still sitting in an open PR. The pin isn’t stale. It’s load-bearing.

So #183 stays on 22.04 and caches the .deb files instead. On a cache hit it skips apt-get update completely and installs offline with dpkg -i. Things I’d point out to a reviewer:

• No third-party action. Plain actions/cache plus apt, after the Trivy wrapper-action compromise earlier this year.
• Acquire::Retries=3 handles a mirror that fails. It does nothing for a mirror that’s just slow, which is what the cache is for.
• Two checks after install (pkg-config --exists webkit2gtk-4.1, and patchelf on PATH), so a bad set of archives fails right there and not as a confusing bundler error five minutes later.
• The cache key ends in -v1. A key can’t hash a literal list of packages, so bump it by hand when the list changes. A stale set of archives is the one way this breaks.

Result: 27 seconds on the run that filled the cache, then 18 and 11, with Cache restored from key: apt-ubuntu-22.04-tauri-v1 in the log. Honestly, that’s about a 30-second saving on a normal day. The real value is that an 862-second day can’t happen again.

#183’s PR also corrected something I’d said the night before. I’d claimed #181 took the build from 738 to 233 seconds. The build was already around 230 in most runs; 738 was the outlier. #181 stops the occasional cold run. That’s real, but smaller than I made it sound.

Windows is the ceiling (#184)

With the Ubuntu outliers fixed, Windows sets the release time on every run. It isn’t one slow step. Every step is slower:

My guess was Microsoft Defender, which scans every file as it’s written, and a build writes a lot of files. #184 made three changes:

1. Defender exclusions for the workspace, ~/.cargo, ~/go, the Go build cache, the Tauri tool cache and the pnpm store. The step is continue-on-error, so if the cmdlet ever goes away the build gets slower, not broken.
2. A Go cache for the sidecar on all four platforms, to win back what turning off setup-go’s cache in #181 had cost. GOCACHE and GOMODCACHE come from go env because Windows puts them under %LOCALAPPDATA% and %USERPROFILE%. A stable key is fine here, since this workflow only runs on release and there’s no PR churn.
3. A cache for the bundler’s tools. Tauri was downloading them on every run: WiX and NSIS on Windows, and linuxdeploy plus its plugins on Linux. macOS needs nothing.

The PR’s estimate was Windows from 632 seconds to roughly 400 to 450.

What actually happened

Every change worked as designed:

• The Defender step logged excluded for all six paths.
• Cache restored from key: tauri-tools-windows-v1, and the report step listed NSIS in the cache.
• The sidecar’s Go cache hit, and “Build mcphub sidecar” dropped from 70 to 75 seconds to 19.

The Windows leg on the next warm release took 583 seconds.

The sidecar saved about 50 seconds and its cache restore cost 27, so it netted roughly 25. That isn’t a new gain over where things were before #181. Setup Go plus the sidecar build took about 60 seconds on Windows when setup-go’s cache hit, about 100 with no cache, and 77 now. Ubuntu is back to 4 seconds of build plus 5 of restore. #184 mostly undid a regression I’d made a day earlier. The bundler cache saved a download I can’t see as its own step. Defender exclusions made no difference I can measure. “Build the app” at 250 seconds sits right in the middle of where it already was. Maybe hosted runners already go easy on scanning, or the slow part is somewhere I didn’t exclude, like the toolcache that setup-node unpacks into. I don’t know yet.

This is the frustrating part. I don’t think building one binary should take four minutes, and nothing I cached or excluded touched that number. Whatever Windows is doing in there, I didn’t find it this round.

Results (honest version)

There have been only two releases since #184, and one of them filled the new caches. So this is one clean sample, not a measurement.

The median barely moved. What changed is the tail. Each of the two slow releases came from something these PRs remove:

• The 30-minute one was evicted Rust caches. #181 stopped PRs flooding the cache store.
• The 25-minute one was a slow apt mirror. #183 means the release no longer talks to the mirror.

If I’m honest about what I wanted, it wasn’t a 13-minute release. It was knowing a release would take 13 minutes. That’s the part that shipped.

Per leg, Oct 8 00:57 (before) vs 05:49 (after), seconds:

Cache usage on Oct 9 was 10.3 GiB across 30 entries, still over the limit. The old per-run Go duplicates (about 7 entries, about 3 GiB) need to age out or be deleted with gh cache delete.

Where the 13 minutes go now

On the last release: about 20 seconds of versioning and tagging, 583 seconds of Windows, then 165 seconds of publish-mcphub and about 50 seconds of the other publish jobs. Two things stack end to end: the slowest build leg, and the publish that waits for it.

Caching won’t get this to 3 minutes. The levers left:

• Overlap publish-mcphub with the builds. It doesn’t need the desktop installers, so it could start right away. That’s up to 3 to 4 minutes off the whole release for a YAML change.
• A bigger Windows runner. It’s the one lever that speeds up the compile itself, but it isn’t free. Larger runners are billed per minute at a rate that scales with core count, and they don’t draw from the plan’s included minutes. Doubling the cores only breaks even if the leg gets twice as fast, and it won’t: the 90 seconds of setup and the cache restores don’t care how many cores there are. It buys wall-clock time for money, which is a fine trade for a release that runs a few times a week.
• Find the Windows setup time. pnpm and Node setup still take about 90 seconds on Windows against about 17 on macOS, and the Defender exclusions didn’t touch it.
• Measure properly. Several releases in a row now that cache usage is under control, with medians per leg. Two runs isn’t enough.

I don’t think there’s a Part 3 here, unless I find a way to stop building the same app four times. A universal macOS build would fold the two Mac legs into one, and building once and having each platform only sign what it needs is the idea I keep coming back to. I don’t know yet if Tauri will let me do that. For now the release takes 13 minutes, and it takes 13 minutes every time.

There’s also Pilum. Release times after Pilum v0.8.0 should drop again, and that one gets its own post. Call it Part 3.