Testing and Verification

"It looked fine" is not verification. A screenshot is a picture of a cache. Verification is a measurement that could have come back negative, and if yours couldn't have, you didn't run one.

The concept

Two different activities get called "testing":

Automated tests, code that checks code. Catches regressions, runs forever, costs time to write and maintain.

Verification, establishing that a specific thing actually works, right now, in reality. Usually manual, usually the thing that finds real bugs.

Startups under-invest in the first and think they're doing the second while actually doing neither. The reason: it's easy to confuse observing something with checking something.

What actually counts as verified

That last row is the gold standard. A verification claim that names what it couldn't establish is trustworthy in a way that an unqualified "done" never is.

⭐ Every cache between you and the truth will lie

This is the most practically useful idea in the chapter. Between your change and what you observe sit multiple layers, each capable of showing you something stale:

 your edit
   └─ build cache          → did it rebuild?
      └─ deploy/install    → did the new artifact reach the device?
         └─ service worker → serving cached assets?
            └─ HTTP/CDN    → cached response?
               └─ app state → stale in-memory data?
                  └─ what you see

Combine two or three and you get a nightmare: you fix a bug, the build silently fails, the old binary installs, the service worker serves old assets, and you conclude your fix was wrong. You're now debugging your code while the problem is entirely in your toolchain, with no signal telling you which.

📐 Before debugging anything, prove your change is running. Add a visible marker, a version string, a log line, a changed colour, confirm you can see it, then debug. Ninety seconds that saves hours.

The pyramid, adapted for a startup

Ordered by value per hour for a small team, which is not the classic ordering:

 ┌──────────────────────────────────────────────┐
 │ MANUAL EXPLORATION                           │ irreplaceable
 │ Use your own product daily, as a user        │ finds "this is dry",
 └──────────────────────────────────────────────┘ dead links, wrong feel
 ┌────────────────────────────────────────────────────┐
 │ END-TO-END ON THE REAL SYSTEM                      │ highest confidence
 │ Drive the flow; inspect the DB; name the row       │ per hour spent
 └────────────────────────────────────────────────────┘
 ┌──────────────────────────────────────────────────────────┐
 │ DATA / CONTENT VALIDATORS                                │ finds what humans
 │ counts · orphans · duplicates · balance · banned patterns│ structurally can't
 └──────────────────────────────────────────────────────────┘
 ┌────────────────────────────────────────────────────────────────┐
 │ UNIT TESTS ON PURE LOGIC                                       │ cheap, fast,
 │ dates, money, permissions, scoring, state machines             │ high value
 └────────────────────────────────────────────────────────────────┘
 ┌──────────────────────────────────────────────────────────────────────┐
 │ CI SMOKE TEST                                                        │ prevents
 │ builds · lints · types · validators · ⚠️ RELEASE-BLOCKING ASSERTIONS │ disasters
 └──────────────────────────────────────────────────────────────────────┘

Unit-test pure logic first. Functions with inputs and outputs and no I/O, date arithmetic, pricing, permission checks, scoring, state transitions. They're trivially testable and they're where silent wrong answers cost most.

Skip UI unit tests early. They're brittle and they test implementation rather than behaviour.

CI: automate the catastrophic checks

Checklists are for things that cost you time. Automation is for things that cost you the business.

- run: typecheck && lint
- run: test
- run: node scripts/validate-content.js       # counts, orphans, duplicates
- run: node scripts/validate-assets.js        # every reference resolves
- name: 🚨 release-blocking assertions
  run: |
    grep -q "DEBUG_MODE = false" src/config.js || { echo "::error::debug flag on"; exit 1; }
    ! grep -rE "(sk_live|SECRET_KEY=)" src/ || { echo "::error::secret in source"; exit 1; }

Anything that would give away your product, leak a secret, or ship a dev flag belongs here, not on a list you read when you're tired.

Testing the untestable

Some things can't be exercised in development, real payments, platform auth, push notifications, hardware.

Mock at the system boundary, not inside your own code. Mocking the payment SDK tests all of your logic. Mocking your own functions tests nothing.

Then, and this is the part people skip, write down what the mock cannot prove. The mock always passes; that's exactly why it's dangerous.

Test the failure paths

Happy paths are the easy 20%. Enumerate deliberately:

• Decline · timeout · offline · slow network
• Empty · maximum · malformed input
• Permission denied
• Returning user · reinstall · second device
• Expired subscription · cancelled mid-period
• Concurrent actions

Test on a clean install. Your dev device has session state, cached data and completed onboarding, which masks the entire returning-user category (Chapters 19, 21).

📐 Best practice

State what would have made this verification fail. If nothing could have, it wasn't one.

Name the row, the number, the specific expected result.

Bypass every cache, cache-bust, hard reload, clean install.

Prove your change is running before debugging.

Unit-test pure logic.

Validate content and data mechanically when you have volume.

Put release-blocking assertions in CI.

Mock at the boundary; document what mocks can't prove.

Keep a "not proven on real hardware" list.

Test failure paths explicitly.

Use your own product daily, as a user.

💀 Common mistakes

Screenshots as evidence. Caches make pictures untrustworthy.

Debugging a fix that isn't running.

"No console errors" as verification. The worst bugs are silent, a paywall granting free access produces none.

Only happy paths.

Only your device. Hides every returning-user flow.

Chasing 100% coverage. Coverage measures lines executed, not correctness.

UI snapshot tests everywhere. Brittle; they break on every design change and catch little.

Mocking your own code. Tests that your mock returns what you told it to.

Trusting a mock as integration proof.

Relying on checklists for catastrophic checks. Four reminders don't turn off a flag; one grep in CI does.

No tests at all on money, dates, or permissions.

The professional workflow

 1. BEFORE DEBUGGING: prove the change is running (visible marker)

 2. FOR EACH CHANGE, ASK: what would prove this WRONG?

 3. VERIFY AT THE SOURCE, cache-busted
    inspect the database · read the file · check the network tab

 4. NAME THE EXPECTED RESULT — the row, the count, the value

 5. TEST FAILURE PATHS from the list

 6. TEST ON A CLEAN INSTALL

 7. AUTOMATE
    unit tests on pure logic · validators on data ·
    CI smoke + release-blocking assertions

 8. FOR UNTESTABLE THINGS
    mock at the boundary · WRITE DOWN what it can't prove ·
    maintain the "not proven on hardware" list

 9. RECORD THE CLAIM HONESTLY
    "Proven: X. Cannot prove without Y: Z."

Tools, websites & costs

Everything essential here is free. Skipping it is a time decision, not a budget one, and unit tests on pure logic plus a CI smoke test are usually under a day combined.

Alternatives & trade-offs

TDD vs test-after. TDD produces better-factored code and slows early exploration. Most startups write tests after the design settles. Both work; no tests on money or permissions doesn't.

Unit vs integration vs E2E. Unit is fast and narrow. E2E is slow, flaky, and tests what users experience. For a small team: heavy unit on pure logic, a handful of E2E on critical paths, little in between.

Manual vs automated. Manual finds new problems; automated prevents known ones from returning. You need both, and early on the ratio should favour manual.

Staging vs production testing. Staging is safe and diverges from production. Feature flags plus careful production testing is often more truthful. Never test payments in production with real cards.

Coverage targets. Useful as a smell (0% on a payments module is a problem), harmful as a goal (chasing 90% produces tests that assert nothing).

Checklist

• I can state what would have made this verification fail
• Verification bypasses every cache
• I proved my change is running before debugging
• Claims name the action, the expected result, and where it was observed
• End-to-end checks inspect the real external system
• Failure paths tested, not just success
• Tested on a clean install
• Pure logic has unit tests, dates, money, permissions, scoring
• Data/content validators run on every change
• CI has release-blocking assertions for catastrophic checks
• Mocks sit at the boundary, and their limits are written down
• A "not proven on hardware" list exists and is being cleared
• I have written "Proven: X / cannot prove: Y"

📓 Case Study: the standard, and the three layers that lied

Project: SOLIS, which had zero automated tests and still shipped a genuinely high-quality product, because its verification discipline was strong even though its testing discipline was absent.

The standard it held, and this is worth copying verbatim:

VERIFIED LIVE end-to-end: Drove the app → the database received: 1 anonymous user, 1 profile, 1 quiz result, and 20 lesson completions (19 via backfill + 1 live via the queue, completing a specific lesson produced row 4:4:pur).

Four properties make that a claim rather than a hope: a specific action (completing one named lesson), a specific expected result (row 4:4:pur), observed in the real external system (the live database, not a mock), and both paths tested, the one-time backfill and the live queue.

Other examples held the same bar: "created a fresh account → deleted in-app → database back to 0 users / 0 rows" verifies deletion by confirming the data is gone, not that the button worked. And the payments work was verified across "fixture products for the US, India (different currency and tiers), a 1-week trial, a paid intro offer, an annual that isn't a saving, and a monthly-only offering", the cases a home storefront never shows you.

💀 The traps: three layers lying simultaneously.

Layer 1, the service worker. For weeks, image work was "verified" by screenshotting the running app. Meaningless: the service worker served cached assets, so a screenshot after a fix could show the old image, a stale fallback, or occasionally the right one. The screenshot was a picture of the cache.

The rule that replaced it:

Verify images by file-read and an in-browser preload with a cache-bust, NOT app screenshots, the service worker serves stale cached art.

const img = new Image();
img.onload = () => ok(img.naturalWidth > 0);   // naturalWidth: did it DECODE?
img.src = path + "?v=" + Date.now();            // ← what makes it truthful

Layer 2, the build. The build script piped its output into grep, so the pipeline's exit status was grep's, not the compiler's. A failed build "succeeded," and the script installed a stale binary from a previous run while printing a success message (Chapter 33).

Layer 3, stale install state. A dev device with existing sessions and data masks every returning-user path.

Stack them and the failure mode is genuinely vicious: fix a bug → build silently fails → old binary installs → service worker serves old assets → screenshot shows old behaviour → conclude the fix was wrong → write a different fix. You are debugging your code while every problem is in the toolchain.

This is precisely why "prove your change is running before you debug" is the first step of the workflow above.

Mocking done correctly, with its limits stated. Real purchases can't run on a simulator, so the purchase logic was verified against a mocked payments object, success, decline, restore-found, restore-not-found. Mocked at the system boundary, so all of the app's own logic was exercised.

And crucially, the limit was written down:

Cannot be proven without a real device: an actual purchase, platform sign-in, and haptics. I'll mark exactly what's proven vs pending.

Validators found what no human could. With 900 generated questions, a mechanical check caught that the stronger answer sat in second position in 294 of 300, a systematic bias invisible in any individual item and fatal in aggregate (Chapter 24).

⚠️ Deviation: the two cheapest layers were missing entirely.

No unit tests on pure logic, despite five obvious candidates, a streak calculation walking backwards through dates, a growth curve that must never exceed its ceiling, a points function that must be monotonic, rank thresholds at exact boundaries, and a hash that must be deterministic and balanced. All pure functions. All load-bearing. An afternoon of work, never done.

No CI at all, which meant no place to put the assertion that mattered most. A development flag that unlocked the entire paid product for free appeared on four separate checklists across the project's documents and was still enabled at the end.

📐 Checklists are for things that cost you time. Automation is for things that cost you the business. Four written reminders did not turn off that flag. One grep in CI would have caught it on every commit for 24 days.

Lessons

1. ⭐ A verification that couldn't have failed isn't one.
2. Name the row. "Completing X produced row 4:4:pur" is a claim; "sync works" isn't.
3. Every cache between you and truth will lie. Screenshots, builds, service workers, install state.
4. Prove your change is running before you debug. Ninety seconds against hours.
5. "No console errors" is not verification. The most expensive bugs are silent.
6. Test failure paths and clean installs. That's where returning-user bugs live.
7. Mock at the boundary, and write down what the mock can't prove.
8. State your limits, "proven X, cannot prove Y" is what makes a status trustworthy.
9. Validators find what humans structurally can't at volume.
10. Automate the catastrophic checks. Four checklists didn't disable one flag; one grep would have.
11. Unit tests on pure logic are an afternoon and cover exactly where silent wrong answers cost most.

Next: Chapter 29: Debugging: A Systematic Method →