Build Pipelines, CI/CD and Deployment

Your build pipeline is invisible infrastructure until the day it lies to you, reports success over a failure, ships a stale artifact, or deploys the wrong branch. Then it costs you a day of debugging a bug you already fixed. The one job of a pipeline is to fail loudly.

The concept

A deployment pipeline turns "code on my machine" into "running for users," repeatably and without you thinking. Three layers:

 CONTINUOUS INTEGRATION (CI)     on every push: build · lint · type · test · validate
        │
 CONTINUOUS DELIVERY / DEPLOY    merge to production branch → deploy automatically
        │
 RELEASE                         make it live (web: instant; mobile: store review)

The value is not automation for its own sake. It's that the same steps run the same way every time, and a failure stops the line instead of shipping.

The cardinal rule: fail loudly

Every step that can partially fail must verify its own post-condition and fail loudly when it isn't met. "The command ran and exited 0" proves it ran, not that it worked. A build that "succeeds" while producing nothing, a deploy that uploads half the files, a sync that copies stale content, all exit zero.

Silent success is worse than loud failure, because you build on top of it for hours or days before discovering it (Chapters 16, 29).

Concrete shell hygiene:

• set -euo pipefail at the top of every script.
• A pipeline's exit status is the last command's, build | grep error returns grep's status, not the build's. Never pipe a command whose failure matters.
• Never || true on anything whose failure you care about.
• After a build, assert both the exit code and a success marker in the log.

CI: automate the catastrophic checks

Checklists are for things that cost you time. CI is for things that cost you the business. Put here anything that, if it shipped, would be a disaster:

- run: typecheck && lint
- run: test
- run: node scripts/validate-content.js     # if content-heavy
- name: 🚨 release-blocking assertions
  run: |
    grep -q "DEBUG_MODE = false" src/config.* || { echo "::error::debug flag on"; exit 1; }
    ! grep -rE "(sk_live|SECRET_KEY=|password *=)" src/ || { echo "::error::secret in source"; exit 1; }

A flag that gives away your product, a secret in source, a dev endpoint, none of these belong on a checklist you read when tired. They belong in a check that fails the build every time (Chapters 28, 30).

Environments

 local        your machine, real dev keys, gitignored .env
 preview      per-branch, auto-deployed, test keys      ← the killer feature
 production   the real thing, restricted, real keys

Preview deployments per pull request are the highest-value modern deployment feature, every change gets a shareable URL you (and testers) can see before it's live. If your host offers them, use them.

The production branch is a decision

Write down which branch deploys to production. Then verify the live site after every promotion. Every "why isn't my change live?" incident is a branch mismatch (Chapter 16), and it's especially costly when the missing thing is a required page (a privacy policy that a store submission depends on).

Rollback

You will ship something broken. Have a way back:

• Web: instant rollback to a previous deployment (one click on modern hosts). Or a feature flag to disable the broken path without a deploy.
• Mobile: you cannot hotfix, the bad build is live until the next passes review (days). This is why mobile testing is stricter and why over-the-air update tools (for JS-layer fixes) are valuable.

Keep a hotfix build ready near a launch. Something breaks; a prepared build turns a crisis into an afternoon.

Mobile build pipelines

Mobile adds signing, provisioning, and a gatekeeper. Script it to one command, and make it fail loudly, the "stale binary installed while reporting success" failure is specific to mobile and especially insidious (case study). Separate scripts for simulator (no signing) and device (real signing), because they genuinely differ.

📐 Best practice

One command from clean checkout to running app.

Fail loudly, set -euo pipefail, assert post-conditions, never || true on what matters.

Never pipe a command whose exit status you need.

Put release-blocking assertions in CI, dev flags, secrets, dev endpoints.

Use per-branch preview deployments.

Document the production branch; verify the live site after promoting.

Keep secrets out of the repo, injected as environment variables.

Have a rollback path, deployment history or a feature flag.

Keep a hotfix build ready near a launch.

Cache dependencies in CI for speed, but never let a stale cache hide a failure.

Script every environment workaround with a comment explaining why.

💀 Common mistakes

A pipeline that reports false success. The most dangerous, it makes every other bug undiagnosable.

Piping the build into grep. The pipeline returns grep's status.

|| true on something that matters.

Deploying the wrong branch. Confident 404s on pages you know exist.

Secrets committed to the repo. Rotate; git history is forever.

No rollback plan. A broken deploy stays broken while you scramble.

Manual deploy steps. You'll deploy less, and shipping less is the real risk.

No CI at all, so no place to put the catastrophic assertions.

Trusting cached CI artifacts that mask a real failure.

Cloud-synced repo folders that rewrite file attributes and break code signing.

The professional workflow

 1. ONE-COMMAND BUILD from a clean checkout

 2. CI ON EVERY PUSH
    build · lint · type · test · validators · release-blocking asserts

 3. PREVIEW DEPLOY per branch

 4. PRODUCTION on merge to the documented branch

 5. VERIFY the live site — every required URL, in incognito

 6. ROLLBACK path confirmed (history or flag)

 7. FOR MOBILE
    scripted build that FAILS LOUDLY · simulator + device scripts ·
    hotfix build ready near launch

 8. POST-DEPLOY smoke check

Tools, websites & costs

A complete pipeline is free at small scale. GitHub Actions plus a modern host gives you CI, preview deploys, and instant rollback for $0.

Alternatives & trade-offs

Managed platform vs self-hosted. Managed (Vercel, Railway) gives you previews, rollback, and zero ops for a premium. Self-hosted (your own server + a deploy tool) is cheaper at scale and costs your attention. Below meaningful scale, managed wins.

Auto-deploy vs manual gate. Auto-deploy on merge is fast and risky if CI is weak. A manual "promote" gate is safer and slower. Auto-deploy to preview, manual (or gated) to production is a common middle.

Monorepo vs polyrepo CI. Monorepo CI can rebuild everything on any change; good tooling (Turborepo, Nx) caches and scopes. Polyrepo is simpler per-repo. Start monorepo with caching.

Trunk-based vs release branches. Trunk-based with feature flags suits continuous deployment. Release branches suit versioned, especially mobile, releases. Match to how you ship.

OTA updates vs store releases (mobile). OTA fixes JS-layer bugs without review, a real advantage, but is constrained by store policy and can't change the app's fundamental purpose. Use it for fixes, not features.

Checklist

• One command from clean checkout to running
• Scripts use set -euo pipefail; no || true on what matters
• Builds assert exit code AND a success marker
• No command whose status I need is piped
• CI runs on every push
• CI has release-blocking assertions (dev flags, secrets)
• Preview deploys per branch
• Production branch documented; live site verified after promoting
• No secrets in the repo
• A rollback path exists
• (Mobile) build script fails loudly; hotfix build ready near launch
• Repo is outside cloud-synced folders

📓 Case Study: a build script that installed a three-day-old app

Project: SOLIS. The most instructive pipeline failure in the book, because it corrupted every other debugging session while it was live.

The bug. The build script piped its output into grep:

xcodebuild ... build | grep -E "BUILD|error:" || true
xcrun simctl install booted "$APP"
echo "✓ SOLIS running"

In shell, a | b exits with b's status. So xcodebuild could fail catastrophically, and if grep matched anything the pipeline "succeeded", and || true swallowed whatever was left. Meanwhile $APP still pointed at the last successful build's artifact. So the script installed a stale binary from a previous run and printed a success message over it.

The failure mode this creates is uniquely horrible: you fix a bug, run the build, see "✓ running," test the app, and the bug is still there. So you conclude your fix was wrong. You write a different fix. Same result. You are now debugging the build system while believing you're debugging your code, with no signal telling you which. (The user's instinct during an unrelated bug, "are you sure you synced the ios app?", was exactly this suspicion, justified.)

The fix:

xcodebuild ... build > "$LOG" 2>&1
XC_STATUS=$?
if [ "$XC_STATUS" != "0" ] || ! grep -q "^\*\* BUILD SUCCEEDED" "$LOG"; then
  echo "✗ BUILD FAILED — NOT installing. The simulator still has an EARLIER build." >&2
  exit 1
fi

Four changes, each doing distinct work: log to a file instead of piping; capture the exit code explicitly; assert both the code and the literal success line; and, the part that matters most, the error message names the consequence ("the simulator still has an earlier build"), which is the difference between a confusing failure and an obvious one.

📐 A build that can report false success is worse than no build. It makes every downstream bug undiagnosable.

Environment workarounds, scripted with their reasons. The project lived in a cloud-synced folder, which caused three separate, cryptic failures, an attribute-stamping quirk that broke code signing, a permissions issue that broke a package manager's cache, and a locale requirement caused by a space in the path. Each fix was three lines and a comment explaining what breaks and why the fix works. On a new machine eight months later, those comments are the difference between a minute and an afternoon.

A near-disaster from a different silent success. A sync step in the pipeline used a cleanup pattern that, because of a platform regex difference, matched nothing, silently, exit code 0. Junk files accumulated in a tracked directory until they were 263 MB, one commit from permanent history. The fix wasn't just correcting the pattern; it was counting what remained and failing loudly if it wasn't zero (Chapter 16).

The branch bite. Legal pages were committed to the integration branch; the host deployed the production branch. The site went live with its privacy-policy URL returning 404, a store-submission blocker, until the branches were merged. Write down which branch is production, and verify the live URLs.

⚠️ Deviation: there was no CI at all. Which meant there was nowhere to put the one assertion that mattered most. A development flag that unlocked the entire paid product for free appeared on four separate checklists and was still enabled at the end of the project.

📐 Four written reminders did not disable one boolean. One grep in CI would have failed the build on every commit for 24 days.

Lessons

1. A pipeline's only job is to fail loudly. A build that reports false success is worse than none.
2. A pipeline returns the last command's status. build | grep returns grep's. This one line cost real time.
3. Assert two things, the exit code and a success marker.
4. Error messages should name the consequence, not just the failure.
5. Any cleanup step must verify its own result. A pattern matching nothing exited 0.
6. Script environment workarounds with their reasons. Comments save you on a new machine.
7. Name your production branch. Every "why isn't it live" is this.
8. Automate the catastrophic checks in CI. Checklists don't disable flags; assertions do.
9. Keep secrets out of the repo, and a rollback path in reach.
10. Cloud-synced folders are hostile to builds. Attribute stamping, code-signing failures, races.

Next: Chapter 34: Web Infrastructure and Landing Pages →