systems

Pilum: From Launch to Production-Ready in 3 Months

By Dan Flanagan · Apr 16, 2026 · Updated Oct 12, 2026 · 10 min

In December 2025, I open-sourced Pilum: a multi-cloud deployment CLI that deploys to Cloud Run, Lambda, Azure, Cloudflare Pages, npm, Homebrew, and Docker Hub from a single pilum.yaml. The announcement post covered the architecture: recipes, ingredients, handlers, wave-based execution.

That was the “it compiles and the tests pass” version.

Three months and 40+ pull requests later, Pilum deploys all of SID Technologies: platform-core, Torch, Statio, every website, every npm package, and itself. This post is about everything that broke between “it works” and “it ships production software.”

(Update: there’s now a seven-months-in addendum at the bottom. It’s mostly about what stopped breaking.)

The Timeline

Dec 2, 2025: First commit. Baseline CLI with recipe system. (#1)
Dec 4: Homebrew release workflow. Pilum dogfoods its own deployment. (#2-#4)
Dec 31: Service graph, --only-changed, file embedding support. The “I need this for real” features. (#22-#24)
Jan 3-12: Documentation and bug fixes. The quiet “oh, this doesn’t actually work” phase. (#25-#29)
Feb 6-7: The big feature sprint: 8 features in 48 hours. Wave deployments, npm recipe, Cloudflare Pages, Azure Container Apps, Cloud Run Jobs, environment variables, JSON output, history command. (#30-#38)
Feb 8-12: The big fix sprint: YAML parsing, package manager issues, build failures, error swallowing, GCP secrets, Cloudflare execution. Everything from the feature sprint broke something. (#42-#49)
Feb 24: Security hardening and npm publishing fixes. (#51-#53)
Mar-Apr: Memory-based worker allocation, wave ordering bug, orchestrator rewrite. The “I thought this was done” phase. (#58-#62)

The pattern is clear: features ship fast, fixes ship faster, and the real bugs show up a month later.

The feature sprint, and the fixes it cost

Wave-based deployment was the headline feature in #31. Services declare dependencies, Pilum builds a dependency graph, topologically sorts it into waves, and executes waves in parallel. It worked perfectly in tests. Then I tried to publish npm packages.

Pilum publishes @sid-technologies/base-ui, which depends on @repo/configs. Both are in the same monorepo. Wave ordering correctly put configs before base-ui. But worker allocation was distributing work across goroutines without respecting wave boundaries, so workers from wave 2 could start before wave 1 fully drained. The npm registry would reject base-ui because configs hadn’t finished publishing yet. Sometimes it worked, a race condition timing thing, sometimes it didn’t.

The fix (#61) was two things: memory-based worker allocation, so Pilum doesn’t spawn more workers than the machine can handle, and a proper wave barrier, so wave N+1 doesn’t start until every worker in wave N has completed and reported success. Getting there meant rewriting the whole orchestrator in #60: runner.go (772 lines) got replaced with execution.go, pipeline.go, steps.go, and waves.go, about 773 lines total, but now testable and correct.

The scariest bug of the sprint (#49) was the worker queue silently swallowing errors from shell commands. A deployment step would fail, the worker would log it, but the orchestrator wouldn’t see the failure, and the deploy would “succeed” with broken services. The command worker’s error channel wasn’t being read in all code paths: if a command failed during the capture phase, reading stdout or stderr, the error got logged but never made it to the result channel. The fix meant propagating errors through every code path, but finding it took 274 lines of new tests on the worker queue and orchestrator first. The bug stayed invisible because services usually deployed fine. It only surfaced when a Docker build failed mid-stream and Pilum reported success anyway. If you’re building a deployment tool, test the failure paths harder than the success paths. Nobody notices when deploys work. They notice when deploys claim to work and didn’t.

GCP Cloud Run expects secrets in a specific format, SECRET_NAME=projects/PROJECT/secrets/NAME/versions/latest. Pilum was passing them as SECRET_NAME=value, which is right for environment variables but wrong for secret references. Two fixes (#48, #58) covered it: a config parser that detects whether a value is a secret reference or a literal and formats accordingly, and environment variable handling that properly separates --set-env-vars from --set-secrets in the gcloud run deploy command. This is the kind of bug that only shows up when you deploy a real service with real secrets, never against mocked GCP APIs.

Cloudflare Pages broke twice (#47, #56). First the recipe itself: the wrangler CLI had changed its flags between versions and the recipe still had the old format hardcoded. Second, more subtle: the capture worker was incorrectly detecting the end of output from the wrangler process, so it would sometimes truncate the deployment URL and Pilum couldn’t report where the site had gone. The fix was a one-line change to how the capture worker detects stream completion, but it took an hour to diagnose because wrangler’s output format is inconsistent between deploy targets.

The YAML parser (#42) had its own encoding bug. Users could write build: go in their pilum.yaml, but the YAML library parsed go as a boolean, true in YAML 1.1, and the build step broke because it expected a string. Two-line fix, forcing a string type on specific fields during parsing. It’s the kind of bug that makes you question every YAML-based config format you’ve ever designed.

Getting npm packages to publish correctly took three separate PRs. #52 added workspace resolution: in a pnpm monorepo, pnpm publish needs to know which workspace you’re publishing, so I added a resolve_workspaces.js script that walks the workspace config and finds the right package. #53 followed because I realized I had no way to test the full publish flow without actually hitting the npm registry, so I built a test harness with fixture packages and dry-run publishing. #57 handled npm vs pnpm command resolution, since some systems have one but not the other, and the completion and local command system needed to detect which package manager was available and use it.

The December launch had the core recipe engine and a few deploy targets. Everything else got added in February because real usage demanded it: wave-based deployments (#31, the headline feature that later broke), the npm recipe (#32, required for the monorepo workflow), Cloudflare Pages (#34, every SID website uses this), Azure Container Apps (#38, because “multi-cloud” means more than GCP), GCP Cloud Run Jobs (#30, for batch processing and migrations rather than just long-running services), JSON output (#36, for CI/CD integration), history and status commands (#37, #40), environment variable support (#33), a recipes command (#45), and GCP Cloud Run connections (#55, VPC connectors, Cloud SQL connections, service-to-service auth). Ten features in three weeks. Each one spawned a bug fix or two in the following weeks.

The GitHub Action, the website, and the security pass

By February 9, Pilum was stable enough that I didn’t want to keep writing curl | sh install scripts in every CI workflow. So I built pilum-action: a GitHub Action that installs Pilum and runs any command.

- uses: SID-Technologies/pilum-action@v1
  with:
    command: deploy
    tag: ${{ github.event.release.tag_name }}

It auto-detects the runner’s OS and architecture, downloads the right binary, and passes through all credentials via environment variables. A simple wrapper, but it cut 15-20 lines of boilerplate from every deployment workflow. It hit a bug immediately: the binary name in the release archive included the version number (pilum-v0.3.1-linux-amd64), but the action was looking for pilum. Two quick fixes (#3, #4), and a third in March when the naming convention changed again. Every SID repo now uses pilum-action@v1 for deployments instead of raw shell scripts.

The pilum.dev website launched the same day as the CLI, December 3: an Astro site with an animated terminal demo, deploy target icons, and full documentation. Over the next three months it trailed the tool: automated tag updates on Dec 5 so the terminal demo always showed the current version (#4), full documentation pages on Jan 3 (#5), new provider icons as deploy targets shipped on Feb 7-8 (#7, #8), docs for the whole February sprint on Feb 11 (#9), a design pass on Feb 16 that took it from “functional docs” to “looks like a real product” (#10), and real favicons on Mar 19 replacing the placeholder (#12). The website was always the last thing updated after a feature shipped. Having docs deploy automatically from the same pilum deploy pipeline would have kept them in sync. That’s a future improvement.

PR #51 was a comprehensive security audit response, and the headline finding was shell injection via pilum.yaml values: if a service name or environment variable contained shell metacharacters like $(rm -rf /), they’d be passed unsanitized to sh -c. The fix introduced a shellutil package with Quote() for POSIX single-quote escaping on string commands, ValidateServiceName() to reject names with shell metacharacters, and SanitizeHeredocValue() to escape $, backticks, and backslashes in heredoc values. I also added symlink traversal prevention, path validation, and safe temporary directory handling. Pilum’s attack surface is small, users run their own configs, but in CI/CD environments where configs might be generated or templates might include untrusted input, these protections matter. Later we added Grype and Syft vulnerability scanning to the CI pipeline (#51 and subsequent): every dependency gets scanned against the NVD database, and builds fail on high-severity CVEs.

The Numbers

Four months of dogfooding (December 2025-April 2026):

62 PRs merged (17 features, 21 fixes, rest chore/docs)
Deploy targets used in production: Cloud Run, Cloudflare Pages, npm, Homebrew (4 of 7)
Repos deploying with Pilum: 6+ (platform-core, Torch, Statio, all websites, Pilum itself)
Orchestrator rewritten: once (the 2000-line runner.go → modular architecture)
Security findings addressed: 11
Feb 7, 2026: 8 features shipped in one day. Feb 8-12: 5 bugs found from that sprint.

What the 62 merged PRs were

	PRs
Features	17 PRs
Fixes	21 PRs
Chore and docs	24 PRs

What I’d Do Differently

E2E tests from day one. Unit tests caught maybe 30% of the bugs on this list. Every serious bug (wave ordering, error swallowing, npm publishing, Cloudflare output) required the full pipeline to reproduce. I built the E2E framework in #53. Should have been #2.
Fewer features per sprint. The Feb 7 feature marathon (8 features in 48 hours) created a week of bug fixes. Shipping 2-3 features with immediate dogfooding would have caught issues faster.
The orchestrator should have been modular from the start. The 772-line runner.go was unmaintainable. The rewrite into execution.go, pipeline.go, steps.go, and waves.go should have been the original architecture. The monolithic runner made every bug harder to find and every fix riskier.

The gap between “open-source announcement” and “tool I trust with production deploys” was about 40 pull requests. Every real deployment surfaced an edge case that unit tests never caught, and the wave ordering bug stayed invisible for two months because most deploys don’t have inter-service dependencies. If you’re building developer tools, dogfood them immediately. The first month after launch is a second development phase, and it’s worth budgeting for.

What’s Next

Pilum is stable. It deploys everything SID Technologies builds. The immediate roadmap: more deploy targets as needed (Fly.io, Railway), a recipe marketplace for community contributions, better error messages (the #1 user experience issue), and eventually Pilum Cloud, a hosted control plane, though that’s a different product, not an open-source feature.

Update: Seven Months In (July 2026)

The most interesting thing that happened to Pilum since April is what stopped happening.

The post above covers December through April: 62 pull requests, an orchestrator rewrite, a security audit, and a feature sprint that took a week of fixes to clean up. April through late July looks nothing like that. 15 commits: three features, ten small fixes, a refactor, and a chore. v0.6 to v0.7.4. No rewrites, no multi-PR bug sagas, nothing new for the “everything that broke” list.

Meanwhile the surface area kept growing. Pilum deploys everything SID Technologies ships: GCP services, every website, Homebrew formulas, and now private npm packages too. All of CI/CD runs through pilum-action. I stopped thinking about the tool. For a deploy tool, that’s the whole job.

Two things from the quiet months are worth writing down. The first is that the deploy config became the dev environment. pilum compose (#65) reads the same pilum.yaml files that deploy to production and generates a docker-compose.yml: services wired with depends_on, healthchecks, and infrastructure dependencies declared once in .pilum.yml. For most of my projects that means a Postgres container plus my services, and it works well. I use it constantly while developing, spinning containers up and down with the same configs that ship.

The one project where it wasn’t simple is where the lesson came from. I have a gateway service that routes to multiple backends, and a dependent project that lives in a different repo. I wanted all of those containers in one stack behind a single command, and I kept hitting port conflicts. The fix wasn’t more generator features. Docker Compose already has an override convention: the generated docker-compose.yml stays regenerable, and a hand-written docker-compose.override.yml carries the extra containers and the port fixes, and Compose merges the two automatically. If your tool generates a file, decide on day one where human edits go, because they’re coming. If the answer is into the generated file, you’ve built a one-way generator.

The second thing is that features started arriving with their failure paths tested. The one substantial feature of this stretch was Cloud Run sidecars plus two image-based recipes for deploying prebuilt images (#72). It shipped with its own e2e fixtures and an end-to-end test runner in the same PR, and spawned zero follow-up fixes. In February, a feature that size cost me two weeks of cleanup. Earlier in this post I said the E2E framework should have been #2 instead of #53. Following that advice this time worked.

The February version of this post ends every section with a bug. The July version doesn’t have the material, because nothing broke badly enough to write about. That’s what stable looks like, and it’s also what boring looks like. Seven months in, I’ll take it.