AI Data Operations —
A PM's Field Guide

In early 2026 I spent several weeks doing deep research into AI Data Operations — specifically the role a Technical Product Manager plays in managing annotation pipelines, RLHF workflows, and quality assurance for AI model development. I did hands-on practice with Labelbox, built annotation guidelines from scratch, and worked through the full operational lifecycle from data ingestion to client delivery.

This document captures what I learned. I am sharing it publicly because I believe the PM community needs more practical, operational knowledge about how AI training data actually gets made — not just the theory, but the day-to-day mechanics of running a production annotation operation. Everything here is my own synthesis.

Mental model: production operations, not project delivery

The most important thing I learned is that AI Data Operations is not a project management role in the traditional sense. It is a production operations management role.

In a traditional PM role I asked: when will it be done? In AI Data Ops I learned to ask: how well is it running right now?

The best analogy I found was a pedal pub. The PM is the main pedaller — setting pace, keeping the vehicle moving, steering through obstacles. Annotators are the other riders contributing power. The PM Data Ops Lead is the driver navigating the route. Nobody gets off while it is moving. Everything must stay in sync or the whole pub wobbles.

The RLHF layer adds a dimension that makes this even more precise — as you pedal, the road itself changes underneath you, because your annotation work feeds back into the model and shifts what the next batch of data looks like. You are simultaneously keeping the vehicle moving and recalibrating your route in real time.

The lifecycle: ADLC vs SDLC

Traditional software delivery runs on an SDLC — a linear cycle with a defined end. Build, ship, done.

AI Data Operations runs on what I call an ADLC — Annotation Data Lifecycle. It is a continuous production loop with no finish line:

Ingest → Annotate → QA Review → Deliver → Report → Ingest again

The batch is the unit of work. Each batch that clears the quality threshold ships to the client. The next batch immediately begins. The flywheel never fully stops — it transitions.

Key ADLC metrics (replacing velocity and burn-down):

The three operating pillars

Every PM Data Ops role organizes around three simultaneous responsibilities — three engines running in parallel, not three phases running in sequence.

IRR and Arbitration — cause and effect

IRR and Arbitration are not two separate processes. They are cause and effect. IRR tells you the temperature of your operation. Arbitration finds the cause and produces the fix.

IRR % = Agreements ÷ Total reviewed × 100

Thresholds I use as reference:

• 90%+ — healthy, guidelines are working
• 80–89% — warning, investigate root cause
• Below 80% — stop, fix before continuing

The three root causes of every disagreement

Every disagreement in annotation traces to one of exactly three causes:

1. Guideline gap — the guidelines did not give both reviewers enough to reach the same conclusion independently. Fix: update the guidelines. This is the PM's fault, not the annotators'.
2. Hard image — the guidelines are clear but this specific image is genuinely ambiguous. Fix: make a final ruling, log it in the edge case section. Guideline does not change.
3. Annotator error — one reviewer clearly misapplied an unambiguous guideline. Fix: annotator calibration conversation, not a guidelines update.

Diagnosing which of the three applies is the most important judgment call in quality operations. Getting it wrong produces the wrong fix and the same problem recurs.

Annotation guidelines — the Managing Editor role

The annotation guidelines document is the source of truth for everything downstream. Structure I use for every defect class:

1. One-sentence definition
2. What qualifies — 6 to 8 specific examples
3. What does NOT qualify — 6 to 8 specific examples with brief rationale
4. Nested attributes — each attribute with all options explicitly defined
5. Dataset-specific edge case rulings
6. Good vs bad annotator note examples

The negative examples in section 3 are the most important part. This is where annotator disagreement lives — not in ambiguity about what a defect is, but in uncertainty about what it is not.

The version control discipline

I separate speed from detail by operating in two modes:

• During live queue: one-sentence ruling in a decision log. Ship it to annotators in under 5 minutes. Keep the flywheel moving.
• End of day: transfer all decision log entries into the proper guidelines document with full context. Polish then, not during.

Batch scoping and SLA — the formula

Batch size is not a gut call. It is a calculation with six inputs:

Input	What it tells you
TPT	How long per image — measure from pilot batch, never assume
Annotator capacity	Productive hours per day — 4 to 5, not 8
QA overhead	IRR review adds 15 to 20% to total annotation time
Image complexity	Multi-defect images multiply TPT — audit before scoping
Client urgency	Their model training schedule is your ceiling
IRR threshold	Higher target = smaller safer batches

(Annotators × Productive hours × 60 min)
÷ (TPT + QA overhead per image)
× 80% buffer
= Committed batch size per SLA window

The 80% buffer absorbs unexpected blockers. If you never use it, you deliver early. If something goes wrong, it saves your SLA.

RLHF — where my team's work fits

RLHF (Reinforcement Learning from Human Feedback) connects annotation work to model improvement. The three operational entry points:

• Stage 1 — Supervised Fine-Tuning (SFT): Annotators write or select ideal responses. Model learns what good looks like. My guidelines document governs quality here.
• Stage 2 — Reward Model Training: Raters compare two model outputs and select which is better. The primacy effect (order bias) is a structural risk — task design must randomize presentation order.
• Stage 3 — PPO Policy Optimization: Mostly automated. My team validates outputs after optimization and flags new failure modes, feeding back into the Red Team prompt library.

The RLHF loop means the flywheel never fully stops. My labeled data trains the model. The improved model generates harder images. Harder images come back to me as the next batch.

Red Teaming — what the PM owns

Red Teaming is structured adversarial testing — deliberately trying to make the model fail before real users do. Two categories:

• Jailbreak datasets — prompts engineered to bypass safety guidelines
• Safety datasets — prompts probing bias, hallucination, tone failures, factual errors, and behavioral inconsistency

As PM I own the library, not the attacks: version control (every prompt has an ID, creation date, model version), tagging by type/category/severity/status, currency (retiring stale prompts), and regression readiness. The prompt library is institutional memory.

Agile in Data Ops — the right framework for each layer

Agile operates at different layers with different purposes:

• Daily production queue: Kanban, not Scrum. Work arrives unpredictably, guidelines change in real time, and there is no defined end state. Sprint time-boxing breaks all three of these realities simultaneously.
• Batch delivery cycle: Agile ceremonies adapted. Sprint planning → batch scoping. Daily standup → queue pulse check. Sprint review → batch delivery review. Retrospective → after-action review.
• Tooling and process improvement: full Scrum. Defined scope, clear done state, prioritized backlog. Short sprints — one week maximum.

The decision rule: Does this work have a defined end state and predictable scope? Sprint. Does it flow continuously with unpredictable arrival and real-time change? Kanban.

Hands-on practice: what I built

To ground this research in operational reality I ran a self-directed simulation using Labelbox free tier and AI-generated images from Canva as my dataset:

• Designed a five-class defect ontology (SPELL_ERROR, FONT_INCONSISTENCY, DATE_NUMBER_ERROR, MISALIGNED_TEXT, LAYOUT_DEFECT) plus a CLEAN class
• Wrote a complete annotation guidelines document with negative examples, nested attribute definitions, and dataset-specific edge case rulings
• Configured the ontology in Labelbox including nested attributes, multi-select checklists, and a global annotator notes field
• Ran a 5-image pilot batch to validate tool configuration
• Used a vision-capable AI as an independent second reviewer to calculate IRR without memory bias
• Built a RAID log, decision log, and TPT tracking spreadsheet
• Produced a mock client status report in "no surprises" format

Where this knowledge transfers

The operational disciplines here are not specific to annotation projects. They transfer to any role where work arrives continuously rather than in planned sprints, quality must be measured against a defined standard in real time, guidelines need to evolve while production continues, and multiple stakeholders need transparency into an operation they cannot directly observe.

These conditions describe a large portion of AI product management work as the field matures — not just annotation, but model evaluation, red teaming operations, data governance, and AI quality assurance broadly.