Product development lifecycle · any AI product, any company

AI PM Playbook

Strategy to sunset as one sequence: who hands what to whom, in order, and the gate that closes each phase. Built to be product- and company-agnostic: consumer, B2B SaaS, marketplace or developer platform. Indigo marks the work an AI product adds on top of a classic PDLC. Every phase has a template, and a worked example fills them all in.

Request / hand-off Response / result AI-specific step Phase gate

Swipe sideways to see all eight roles →

01StrategyCompany mission, OKRs, AI thesisMarket research: segments, TAM, unmet needsMarket signals: switching triggers, willingness to payMarket + competitor scan: where AI is a moat vs. a featureInventory proprietary data + signalsData readiness + build / buy / fine-tune optionsStrategy memo: where to play, how to win, 2–3 AI betsGATE Strategy approved: bets + investment envelope02VisionVision narrative + 3-year north starConcept prototype / vision videoNorth Star Metric + 3–5 input metrics (metric tree)Autonomy ladder: assist → copilot → agentVision buy-inGATE North Star Metric, input metrics and target autonomy agreed03DiscoveryCustomer research: interviews, JTBD, surveys, shadowingFunnel + cohort analytics: where users drop and whyPains, tolerance for AI error, what builds trustWizard-of-Oz / prompt prototype on real samplesBaseline quality on 50–100 real casesEarly risk screen: PII, regulated decisions, IPRisk tier + constraintsGATE Problem worth solving AND AI is the right tool04RoadmapSize opportunities: RICE + model cost and latencySizing request incl. inference + infra costEstimates + platform dependenciesNow / Next / Later roadmap of outcome betsFunding + headcountRoadmap preview for pricing + packagingGATE Quarterly roadmap committed05DefinePRD: problem, personas, non-goals, scopeSuccess metrics: KPIs, targets, guardrail metricsInstrumentation plan: events, dashboards, A/B setupEval spec: golden set, rubric, quality bar, failure taxonomyOffline eval harness + baseline scoreAI UX: confidence cues, citations, undo, human hand-offGuardrail reqs, data-use terms, model card draftAcceptance criteria: p95 latency, cost per taskGATE PRD + eval spec signed: launch bar set before build06BuildRetrieval, tools, prompts or fine-tune pipelineEval run: quality, hallucination rate, costTriage failures → data, prompt, or model fixUsability tests on working prototypeTask success + trust findingsObservability: traces, feedback capture, kill switchGATE Offline eval ≥ launch bar on 2 consecutive runsloop ↻07Trust reviewRed team: jailbreaks, prompt injection, bias, PII leaksMitigations + guardrail test resultsSign-off with documented residual risksGATE Responsible-AI sign-off08LaunchPositioning, pricing, enablement, support runbookDogfood → 1% → 10% → GA, A/B vs. controlKPIs: activation, adoption, retention, conversionAI signals: acceptance rate, overrides, escalationsLaunch comms + AI disclosureLaunch readout: NSM + KPI deltas vs. controlGATE Go / no-go at each ramp step: guardrail metrics hold09Operate + iterateDrift, cost, latency, incident alertsThumbs, corrections → new golden-set casesNew model versions re-scored on golden setWeekly metrics review: NSM, KPIs, guardrailsContinuous research: NPS / CSAT, churn interviewsRetro: update roadmap + eval setQBR: NSM trend, KPIs, ROI, cost per task, risk logDouble down / pivot / sunset → next strategy cycleloop ↻ LeadershipLeadershipAI PMAI PMUsersUsersDesign / UXDesign / UXEngineeringEngineeringData / MLData / MLLegal · TrustLegal · TrustGTM · SupportGTM · Support
Nine phases, eight roles. The AI PM lane carries most traffic because the PM is the router between strategy, users, model quality and risk. Build and Operate repeat until their gate holds; Operate feeds the next Strategy cycle.

Phase by phase

#PhaseAI PM's jobArtifactsKey metricsPitfall to avoidTemplate
01StrategyPick where AI creates a durable advantage, tied to company OKRsStrategy memo, market sizing, AI bets, data-moat mapTAM / SAM, proprietary data coverage, build vs. buy TCOStarting from 'we need AI' instead of a customer or business problemAI strategy memo
02VisionPaint the end state and how much autonomy the product earns over timeVision narrative, concept prototype, North Star Metric + metric treeNSM and its input metrics; target autonomy per workflowA vision with no measurable North Star, or a vanity metric as the North StarNorth Star + metric tree
03DiscoveryProve the problem is real and that AI beats a rules-based fixResearch synthesis, personas, JTBD map, funnel analysis, risk tierBaseline quality on real samples; user error toleranceDemo-driven conviction: testing on 5 cherry-picked examples instead of real dataDiscovery kit
04RoadmapRank bets by value, effort, model cost and riskNow / Next / Later roadmap, sizing, funding askCost per task and latency in the prioritization scoreIgnoring inference cost until after launch, when margins breakRoadmap prioritization Unit economics
05DefineWrite the launch bar before anyone buildsPRD, KPI targets, instrumentation plan, eval spec + golden set, model card draftQuality bar, hallucination ceiling, p95 latency, cost per taskStarting build with no eval set, so 'good enough' is argued, not measuredAgent PRD Eval plan GenAI + agentic metrics
06BuildRun the eval loop and triage every failure to its root causeEval dashboard, failure log, instrumented prototypeOffline eval score trend across iterationsTuning prompts by vibes; fixing one failure and silently breaking three othersExperimentation guide
07Trust reviewMake sure the system fails safelyRed-team report, mitigations, sign-offJailbreak / injection pass rate, bias checks, PII leakageTreating red-teaming as a launch checkbox instead of a test suite you keepResponsible AI review
08LaunchRamp in steps and let guardrail metrics decide each stepGTM brief, rollout plan, KPI dashboard, A/B readout, AI disclosureActivation, retention, acceptance and override rate vs. controlGoing 0 → 100% in one step with no guardrail metrics or kill switchLaunch checklist
09Operate + iterateTurn production feedback into eval cases and roadmap changesMetrics review, NPS / CSAT, QBR deck, updated golden setNSM trend, drift, cost per task, incidents, ROIShipping and moving on while the model drifts and costs creepPost-launch runbook

The metrics stack

Every AI product needs all five layers. The North Star says if you are winning; guardrails say if you are winning the wrong way.

LayerWhat it answersConsumer appB2B SaaSMarketplaceDeveloper platform
North Star MetricIs the product delivering its core value?Weekly tasks completed with AI helpWeekly accounts where AI completes a workflowSuccessful matches per weekWeekly active builders shipping with the API
Input metricsWhich levers move the North Star?New users, % using AI weekly, tasks per userSeats activated, workflows automated per accountSupply listed, search-to-match rateTime to first call, calls per active key
Business KPIsIs it working as a business?D30 retention, paid conversion, ARPUNRR, expansion, churn, CAC paybackGMV, take rate, repeat ratePaid conversion, usage revenue, churn
AI quality metricsIs the model good enough to trust?Answer acceptance, edit rate, hallucination rateTask success, accuracy vs. golden set, human-override rateRanking relevance (NDCG), match qualityEval pass rate, tool-call success, p95 latency
Guardrail metricsAre we winning the wrong way?Unsafe output rate, complaints, cost per taskEscalations, data-leak incidents, cost per taskFraud, bias across segments, cancellationsError rate, abuse, cost per 1K calls

GenAI and agentic metrics: the reference set

The metrics stack above says which layers you need. These are the GenAI- and agent-specific metrics inside them. Pick one North Star, three to five inputs, a guardrail for each, and the layers your product type needs. Picks for eight product types, formulas, eval statistics, an event schema and a weekly review table: GenAI and agentic AI product metrics template.

LayerCore metricsTrap to watch
Business valueIncremental revenue vs. holdout, cost-to-serve reduction, measured time saved, paybackComparing adopters to non-adopters instead of a holdout
North Star + adoptionTasks completed with output users kept, activation (first success), repeat usage, retention curveActivity metrics (prompts, tokens, chat time) rise when users are confused
User-perceived qualityAcceptance rate, edit distance after accept, regenerate rate, override rate, CSATAcceptance is not accuracy
Reliance + teamingOver-reliance (wrong outputs accepted), under-reliance, verification time, human + AI vs. human alone, sycophancyRising trust with flat accuracy means rubber-stamping
Model quality (offline evals)Task success, pairwise win rate, faithfulness, factuality, top-harm error, calibration, pass^k, regression rate, judge agreement, golden label qualityThresholds set after results; gating on pass@k instead of pass^k
Retrieval + contextHit rate@k, recall@k, precision@k, MRR, NDCG, context position sensitivity, truncation rate, citation accuracy, answerable-but-refused, unanswerable-but-answered, index lagLow hallucination rate built on refusals
Task outcomesContainment with resolution and false containment, automation and straight-through rate, revert rate (coding), keep rate (generation), source coverage (research)Containment reported alone counts giving up as a win
Agent behaviorVerified task completion, claimed vs. verified gap (silent failure), wrong-tool rate, steps per task, loop rate, correct + unnecessary handoff, irreversible-action errors, undo rateTrusting the agent's own “done”
Agent permissions + securityUnauthorized actions, least-privilege coverage, destructive actions blocked, indirect prompt injection, tool-description poisoning, permission-boundary leakage, credential exposureTesting only direct attacks, not injection through documents and tool results
Multi-agentToken multiplier and quality gain vs. single agent, handoff failures, duplicated work, orchestrator overheadNo single-agent baseline
Memory + personalizationWrong or stale memory, memory helpfulness, cross-user leakage, deletion complianceDeleting the memory store but not derived indexes
Trust + safetyUnsafe output, jailbreak success, data leakage, over-refusal, fairness gap, disclosure + content marking (EU AI Act Art. 50)A UI notice without machine-readable marking
PerformanceTTFT, output tokens/sec, p95 end-to-end latency, time to task completion, fallback rateReporting only p50
Voice + multimodalWord error rate, voice-to-voice latency, barge-in success, MOS, image/document accuracy, prompt adherenceWER measured on clean audio only
Cost + unit economicsBlended cost per successful task, quality per dollar, tokens per task, cache hit rate, model mix, review cost, gross marginDividing cost by all requests makes failures look cheap
LLMOps healthProduction quality score, offline–online agreement, quality and input drift, golden-set freshness, feedback-to-eval time, MTTD / MTTRA frozen golden set while user behavior moves on

Customer research by phase

PhaseQuestionMethodsOutput
StrategyWhich market and segment are worth it?Market sizing, analyst + competitor teardown, win/loss reviewTarget segments, TAM / SAM
VisionWhat would ideal look like for the user?Co-design sessions, concept tests, diary studiesVision prototype users react to
DiscoveryIs the problem real and frequent?Interviews, JTBD, surveys, contextual inquiry, funnel + cohort data, support-ticket miningPersonas, opportunity tree, problem ranking
RoadmapWhich problems matter most?Kano / MaxDiff surveys, sales + support input, usage dataRanked opportunities
DefineDoes our solution concept fit?Wizard-of-Oz, fake-door tests, prototype walkthroughsValidated scope and AI UX
BuildCan people use it and trust it?Moderated usability, think-aloud, beta cohortFixes to UX, trust cues, hand-off
LaunchDoes it change behavior at scale?A/B tests, in-product surveys, session replaysKPI deltas, adoption blockers
OperateWhat keeps users, what loses them?NPS / CSAT, churn interviews, feedback tagging, logs of overridesRoadmap inputs and new eval cases

What makes it an AI PDLC