AI Integration

An AI feature is the first thing most founders build that has a variable cost per user. That single property changes your unit economics, your abuse surface, your latency budget, and what "correct" even means. Decide all of it before you build.

The concept

Adding AI to a product is usually straightforward engineering, an API call. What's hard is everything around it:

The single most common mistake is not computing cost per user before building. An uncapped AI feature on a free tier is an unbounded liability with a marketing budget attached.

Do the arithmetic first

Before you write anything:

 cost per action  = (input tokens × input price) + (output tokens × output price)
 actions/user/mo  = realistic usage, not best case
 cost/user/month  = cost per action × actions per month
 gross margin     = (revenue per user − cost per user) ÷ revenue per user

Run it for a heavy user, not an average one, heavy users are who break you. Then ask: at what usage level does this user become unprofitable, and what happens at that point?

If the answer is "nothing happens, they keep going," you have an unbounded liability.

Model selection: tier your work

Model families span roughly two orders of magnitude in cost. Using a frontier model for everything is the most common source of AI cost overruns.

A tiered architecture routinely cuts AI spend by 70-90% versus using the top model everywhere. Route by task, not by habit.

When building on Claude, default to the current generation, Opus 4.8 for the hardest reasoning, Sonnet 5 for the bulk of production work, Haiku 4.5 for high-volume cheap tasks. Model IDs change; read the current API reference rather than hard-coding what you remember.

Latency changes the UX

A 3-10 second response is a different product from a 200ms one:

• Stream tokens. Perceived latency drops enormously even though total time is unchanged.
• Show what's happening. "Reading your document…" beats a spinner.
• Make it interruptible. Users must be able to stop.
• Do it async where possible. Generate in the background, notify when ready.
• Cache aggressively. Identical inputs → identical outputs, served free.

Failure modes are different

Traditional software fails loudly. Models fail plausibly:

Prompt injection deserves emphasis. If your product feeds user content, web pages, or documents into a model, that content can contain instructions. Model output must never be executed as a command, used to authorise an action, or trusted to have followed your rules. Validate on the way out.

Evaluation

You can't unit-test "is this good," so build an eval set: 20-50 real inputs with known-good outputs, run on every prompt or model change.

Three levels, in order of usefulness:

1. Assertions, did it return valid JSON? contain the required field? stay under length? Cheap and catches most regressions.
2. LLM-as-judge, a model scores outputs against a rubric. Good for tone and helpfulness; needs its own validation.
3. Human review, a sample, regularly. Nothing replaces reading the outputs.

Log every prompt and response (with consent and retention limits). Your production logs are your best eval set.

Cost controls that must exist before launch

• Per-user rate limits and daily caps
• Max tokens on every call
• Hard spend alerts at the provider
• Tiering by plan, free tier gets the cheap model, or nothing
• Caching for repeated inputs
• A kill switch to disable the feature without a deploy

Privacy and compliance

• Tell users their input goes to a third party. It belongs in your privacy policy (Chapter 40).
• Check the provider's data-retention and training terms. Enterprise tiers usually offer zero-retention.
• Don't send secrets or regulated data (health, payment, children's data) without checking your obligations.
• Regional processing may be required for EU users.

📐 Best practice

Compute cost per user per month before building.

Tier your models by task.

Stream, and show progress.

Use structured outputs, not prompt-begging for JSON.

Build an eval set of 20-50 real cases before you start tuning.

Version your prompts and treat changes as deploys (Chapter 24).

Pin model versions; test before upgrading.

Cap everything, tokens, requests per user, daily spend.

Never execute model output as a command or use it to authorise.

Ground answers in your own data where factuality matters.

Design the fallback for when the provider is down.

Set a kill switch.

💀 Common mistakes

Not computing cost per user. The dominant AI startup mistake.

Frontier model for everything. 10× the bill for tasks a cheap model does identically.

No caps. One user, one script, an enormous invoice.

Free tier with uncapped AI. You are funding strangers' experiments.

Spinner instead of streaming. Feels broken.

Asking politely for JSON. Use structured outputs.

No eval set. You cannot tell whether a prompt change helped.

Unpinned model versions. Your product changes without a deploy.

Trusting model output as instruction. Prompt injection.

Sending regulated data to a third party without checking obligations.

Hiding that AI is involved. Users find out, and it reads as deception.

Building AI because it's expected, into a product that doesn't need it.

The professional workflow

 1. IS AI THE RIGHT TOOL? What breaks if you use rules or search instead?

 2. ⭐ COST MODEL — for a HEAVY user, not an average one
    → cost/user/month → gross margin → where do they become unprofitable?

 3. TIER THE WORK — cheap model where it suffices

 4. BUILD THE EVAL SET (20-50 real cases) BEFORE tuning

 5. IMPLEMENT
    structured outputs · streaming · timeouts · retries ·
    pinned model version

 6. CONTROLS
    max tokens · per-user limits · daily caps · spend alerts ·
    plan tiering · kill switch

 7. SAFETY
    treat all retrieved/user content as DATA · validate output ·
    never authorise from model output

 8. PRIVACY
    disclose in policy · check retention/training terms · region

 9. INSTRUMENT
    cost per call · latency · failure rate · user acceptance

10. RUN EVALS ON EVERY CHANGE — prompt, model, or provider

Tools, websites & costs

Budget honestly: a chat-shaped feature with moderate use typically lands between $0.50 and $5 per active user per month depending on model tier and volume. Model that against your price before you build.

Alternatives & trade-offs

Hosted API vs self-hosted. Hosted is faster, better, and a per-token cost. Self-hosting removes marginal cost and adds GPU spend, ops, and lower quality. Self-host when volume is high, tasks are narrow, and quality requirements are modest.

One provider vs a router. A router gives failover and easy model switching at the cost of a layer and a margin. Worth it once AI is core to your product.

RAG vs fine-tuning vs long context. RAG is the default, cheap, updatable, citable. Fine-tuning suits consistent format or tone at volume. Long context is simplest and expensive per call. Start with RAG.

AI-first vs AI-assisted. AI-first products live or die on model quality and get commoditised as models improve. AI-assisted products use models to remove friction from something valuable on its own, more defensible.

Do you need AI at all? Search, rules, templates and heuristics are cheaper, faster, deterministic and testable. Use AI where genuine language understanding or generation is required, not as a feature checkbox.

Checklist

• I computed cost per user per month for a heavy user
• I know where a user becomes unprofitable, and what happens then
• Work is tiered across model sizes
• An eval set of real cases exists
• Structured outputs, not JSON-by-request
• Responses stream, with visible progress and interruption
• Max tokens, per-user limits, daily caps, spend alerts
• Model versions are pinned; evals run before upgrading
• Retrieved and user content is treated as data, never instruction
• Model output is never used to authorise an action
• AI use is disclosed in the privacy policy; retention terms checked
• Cost, latency and failure rate are instrumented
• A kill switch exists

📓 Case Study: costing a feature, then not building it

Project: SOLIS. This case study is about a decision not to ship, which is the more instructive outcome.

The planned feature. The product plan's Phase 2 was an AI mentor: users write a nightly reflection, and a persona replies in the voice of the person they're becoming. The research had identified this as the strongest differentiator available, the one mechanic no competitor occupied.

⚠️ It was explicitly deferred out of v1, and the reasoning is a good template.

They did the arithmetic before building. From the plan:

AI cost control: cheap model for routine replies, premium model for weekly letters; daily caps on free tier (zero AI), generous-but-capped on premium.

Premium user: ~30 replies + 4 letters/month. With model tiering, well under $1/user/month against $4.17-9.99 revenue, healthy margins.

Everything this chapter recommends is in those two sentences: tiering (cheap model for volume, premium for the rare high-value output), plan gating (free tier gets zero AI, so free users cannot generate cost), caps even on paid, and an explicit margin calculation against actual revenue.

That analysis is what made the deferral a decision rather than a delay. It established that the feature was economically viable, and that it belonged in a later phase because it needed a backend, cost controls, and safety work that didn't exist yet.

The safety requirement that made it Phase 2, not v1. The plan named crisis handling as launch-blocking for the AI feature specifically:

Crisis detection on reflections → supportive response + resources; clear "this is not therapy" framing; age gating.

And a tone constraint written as a hard rule:

The Mentor is aspirational and compassionate, a mentor, never a critic. An idealized self that judges harshly reinforces negative self-talk; tone rules go in the system prompt and are tested.

For a product whose audience included people dealing with low mood, an AI that responds to a written confession is a genuine safety surface. Shipping it without crisis detection, tone testing, and an eval set would have been reckless. Recognising that a feature's safety requirements exceed your current capacity is a legitimate reason to defer it, and far better than shipping it and hoping.

What was built instead is worth noting as the cheaper alternative: 900 hand-authored applied questions, two per lesson, with fixed answers and explanations. Deterministic, zero marginal cost, testable, no safety surface, no vendor dependency, and it delivered a large part of the "applied practice" value the AI feature was meant to provide.

📐 The generalisable move: ask what non-AI mechanism captures most of the value. Authored content, templates, rules and search are deterministic, free at the margin, and testable. Use AI where genuine language understanding is required, not as the first tool you reach for.

🚩 Unvalidated, and honestly so. The AI mentor was never built, so none of the cost modelling was tested against real usage. The estimate of ~30 replies per user per month is an assumption; heavy users could easily be 5-10× that, which is exactly the scenario this chapter says to model. The method is sound; the numbers are untested.

The transferable structure of the decision:

Lessons

1. Compute cost per user before you build. For a heavy user, not an average one.
2. An uncapped AI feature on a free tier is an unbounded liability.
3. Tier your models. Cheap for volume, frontier for the rare hard case, routinely 70-90% cheaper.
4. Gate AI by plan. Free users generating cost is a decision, not a default.
5. Ask what non-AI mechanism captures most of the value. Authored content is deterministic, free at the margin, and testable.
6. Safety requirements can exceed your capacity, that's a legitimate reason to defer, and better than shipping and hoping.
7. Latency changes the product. Stream, show progress, allow interruption.
8. Model output is data, never instruction. Never authorise from it.
9. Build the eval set before tuning, or you can't tell whether a change helped.
10. Pin model versions. Otherwise your product changes without a deploy.

Next: Chapter 24: Prompt Engineering →