AI Product Modernization Playbook · 10 / 12
Computer vision adds value when an image leads to a useful next step: a defective component is diverted, a customer identifies the right device, or a reviewer receives better evidence. Building that capability means designing capture, decisions and feedback together.
Consider a customer trying to set up a router. A photograph could replace a long conversation about its shape, ports and model number. Yet the feature still fails if glare hides the label, the system confuses two similar models, or the application launches the wrong instructions without confirmation. A manufacturing inspection station faces the same pattern at a different speed: seeing an object is only the beginning of getting the action right.
This chapter of the AI Product Playbook explains how to turn images and video into dependable product workflows. It covers controlled inspection and customer photo flows, with separate treatment for identity and safety applications whose consequences demand stronger evidence and controls.
On this page
1. Choose one visual decision to improve
Start with a task someone already performs or a documented gap in the current process. Identify a device, locate a defect, read a meter, retrieve a similar product or prepare a damage review. Define the unit of work: an image, a physical item, a customer session or an event across multiple video frames. The evaluation denominator must match that unit.
Specify the action and the error costs
Write down what follows each result, who owns it and what happens when evidence is inadequate. A product tag can often be corrected cheaply. An incorrect rejection can waste material or exclude a customer. A missed defect can escape into a shipment. These outcomes need different thresholds, review arrangements and release criteria.
Bound the operating conditions: supported product families, defect sizes, camera positions, lighting, line speeds, phone capabilities and locations. A feature that works for one station or catalogue has not automatically been validated for another. Make unsupported objects and conditions explicit states rather than forcing every image into a known class.
Compare against the current alternative, including simple fixtures, barcode scanning, better forms and human inspection. The business measure might be fewer escaped defects at an acceptable false-reject rate, lower cost per completed verification, or more customers finishing setup. A higher benchmark score is useful only when it supports that outcome.
2. Prove that the capture process contains the evidence
Test the camera before investing heavily in model development. Capture real examples at production speed and under expected lighting. Inspect focus, glare, motion blur, resolution, occlusion and whether the relevant surface is visible. For physical measurements, establish scale, lens distortion and calibration uncertainty; a segmentation mask alone does not establish an accurate dimension.
Treat image quality as an operating condition
A surface camera cannot reliably reveal a hidden internal defect without relevant evidence. Depending on the task, another view, depth sensor or different imaging modality may be necessary. Human difficulty seeing a signal is a warning to investigate, not a mathematical proof that a model cannot detect it. Demonstrate that the signal is present and repeatable.
Make capture quality observable. Monitor exposure, sharpness, framing, dropped frames, calibration status and camera connectivity. Define whether a failed check requests another photo, routes an item to inspection or pauses the relevant process. A blank frame must never be interpreted as a defect-free product.
In a 2020 study, Google researchers observed and interviewed nurses across eleven clinics in Thailand using a diabetic-retinopathy screening system. They found tension between the system's image-quality requirements and the data available in resource-constrained clinical settings. The published study supports evaluating the surrounding workflow alongside accuracy; it does not establish that every vision deployment will face the same failure rate.
3. Build a dataset that represents the operating environment
Collect examples across cameras, shifts, suppliers, materials, seasons and user devices that matter to the intended use. Include acceptable variation, not only pristine examples and obvious defects. Record the item or session identifier, capture conditions, source, timestamp and permission to use the material for training or evaluation.
Give the taxonomy an accountable owner
Quality specialists or domain owners should define each label with reference examples, boundary cases and an explicit unknown category. Separate the observed condition from severity and disposition: a visible scratch is an observation; whether it warrants rejection depends on the specification. Version those definitions so a policy change does not silently make old labels inconsistent.
Measure disagreement on a shared sample and adjudicate difficult cases. Disagreement can reveal poor images, ambiguous standards or insufficient context. It limits confidence in evaluation against those labels, but it is not an absolute ceiling on model performance against better-established ground truth. Preserve unresolved ambiguity instead of manufacturing certainty through majority voting alone.
Prevent near-duplicate leakage
Keep frames from the same video, views of the same item and repeated photos from a session together when splitting data. Use later production periods and held-out stations or devices to test the kinds of change expected after launch. Randomly separating adjacent frames can produce excellent test results while measuring little beyond recognition of the same scene.
Use synthetic defects or generated images to explore gaps, with their origin clearly recorded. Validate on real examples held outside development. Synthetic artifacts can become shortcuts, and rare failure mechanisms may be hard to simulate faithfully. Keep development samples separate from a frozen evaluation set and a representative stream of fresh audits.
4. Match the model to the output the product needs
Choose the simplest approach that can meet the task's evidence, accuracy and operating requirements. A deterministic image-processing rule may be adequate for a controlled measurement. A specialized learned model may handle visual variability. A vision-language model can help explore open-ended recognition, but its fluency should not be mistaken for reliable localization or measurement.
Swipe horizontally to read all columns.
| Needed output | Candidate approach | Validation focus |
|---|---|---|
| Item category | Classifier or pretrained embeddings | Confusable classes, unknown items and realistic prevalence |
| Location and count | Object detector, with tracking for video | Small or occluded objects, duplicates and missed events |
| Shape or affected area | Segmentation, with calibrated geometry when needed | Boundary errors and downstream measurement tolerance |
| Unusual appearance | Anomaly inspection using acceptable examples | Harmless variation, rare defects and actionable thresholds |
| Similar products | Visual retrieval with catalogue filters | Useful top results, permissions and current inventory |
| Flexible interpretation | Vision-language model with constrained outputs | Unsupported claims, grounding, latency and cost |
The PatchCore research illustrates inspection built from nominal, non-defective examples using a memory of local image features. Such methods are useful candidates when defect labels are scarce. Their benchmark results do not guarantee performance on a new line, and a visual anomaly is not automatically a rejectable defect.
Use general models for prototyping where appropriate, then benchmark alternatives on the actual task and hardware. Human-check any model-assisted labels. A smaller model is not automatically sufficient, and a larger model is not automatically more reliable. The required evidence should decide.
5. Separate perception from policy and execution
A useful architecture connects capture and quality checks to inference, then applies a separately versioned decision policy. That policy considers the task, permitted actions, data validity and review capacity. The product records both the model output and what actually happened.
Preserve enough context to investigate a decision
Link each inference to its item or session, capture time, camera configuration, preprocessing, model version and taxonomy. Store predicted classes, boxes or masks, score semantics, quality flags, policy version, proposed action and actual disposition. Record whether a reviewer corrected the result and how the final outcome was established.
Retain images only where the purpose and retention policy justify it. Use controlled references, access restrictions and deletion schedules; keep backups and exported datasets within that policy. A model version number is not enough for replay if the image transformation or calibration configuration has disappeared.
The central data service manages labeling, evaluation, model artifacts and staged releases. The local system should remain within its approved behavior when connectivity fails. Edge execution can reduce network dependence and data transfer, but it does not remove the need for security, monitoring or privacy controls.
6. Design for the full physical decision deadline
On a production line, the relevant deadline is when an action must occur on the correct item. Budget for exposure, transfer, preprocessing, inference, queueing, controller communication and actuator response. Test tail latency under realistic load, temperature and power conditions. A fast model in isolation can still miss the operating deadline.
Maintain the link between image and item
Use reliable triggering and item tracking. Account for conveyor movement, missing frames, duplicate triggers, reordering and processing delays. Verify that a reject signal reaches the intended object and that the equipment reports the resulting action. Late results need a defined disposition, not silent application to the next item.
John Deere's See & Spray Select provides a concrete example: boom-mounted cameras scan fallow ground, processing identifies weeds, and selected nozzles apply spray. The product lesson is the complete perception-to-actuation chain. Its operating conditions and vendor-reported benefits should not be transferred to a different crop, station or product without local evidence.
Validate any quantized model or runtime conversion again, particularly on rare classes and threshold boundaries. Batch processing only where it fits the deadline. Specify the response to overload, thermal throttling and unavailable hardware, including what happens when the manual inspection route is already full.
Keep machinery safety functions independently engineered and assessed. A general vision feature may request an action, but should not be assumed to replace a guard, interlock or certified safety function. Test the complete equipment integration with the responsible safety specialists.
7. Make photo capture and correction easy for customers
In a customer application, guide the photograph with framing instructions and useful quality feedback. Explain the fix: move closer, show the label, reduce glare or include the missing view. Offer accessible alternatives for people who cannot complete the camera flow. Do not trap them in repeated retakes.
Confirm consequential interpretations
Show the proposed device or object with a clear confirmation or correction step before selecting instructions or preparing an action. Similar models may require another observable detail. Track wrong confirmations and downstream completion, because a user can accept an incorrect suggestion. Confirmation is a control, not proof of correctness.
Return structured observations with their evidence, quality flags and allowed next step. Keep model observations separate from generated explanations. A highlighted region helps a reviewer find the relevant content, but it does not establish that the explanation is faithful or that the prediction is correct.
For images containing text, treat that text as untrusted input. A photographed instruction must not authorize tools or change application policy. Validate model output against an allowlist of actions and use a separate permission check before any write. The document processing chapter covers the related extraction and verification workflow.
Measure the complete experience: successful capture, retakes, completion, corrections, handoffs and time to resolution. Decide retry limits from observed usability and consequences. A manual route should preserve relevant context so the customer does not have to start again.
8. Treat people, identity and privacy as distinct design decisions
A camera intended to inspect products can also capture employees, bystanders, addresses or number plates. Minimize the field of view and collection at source. Where redaction is appropriate, test its failures; blurring selected regions does not guarantee anonymity when context still identifies someone. Local processing and rapid deletion reduce exposure but do not erase the fact that processing occurred.
Establish the purpose before collecting images
Assess lawful basis, notices, access, retention, vendor processing and whether an impact assessment is required. Ordinary photographs and biometric recognition are not interchangeable legal categories. The UK regulator's biometric guidance explains the role of technical processing and unique identification in determining special-category biometric data. Apply the rules for the actual jurisdiction and use.
The European Commission identifies prohibited AI uses including untargeted facial-image scraping and certain biometric categorization; workplace and education emotion recognition has narrow medical or safety exceptions. Its AI Act guidance is a starting point for checking scope. Do not repurpose a quality camera for employee assessment or identity tracking without a new legal and product assessment.
Evaluate the complete identity workflow
For identity verification, distinguish one-to-one matching, document authenticity, presentation-attack detection and resistance to injected camera feeds. Passing one check does not establish the others. Evaluate the exact pipeline across devices and relevant demographic groups, with error rates, sample sizes and uncertainty. NIST's face recognition evaluations distinguish matching errors and show why image quality and demographic differentials deserve separate analysis; an algorithm result is not certification of an entire application.
In the Rite Aid matter, the FTC alleged inadequate safeguards and harms from incorrect facial-recognition matches. The 2024 stipulated order included a five-year restriction on covered facial-recognition use; the defendants neither admitted nor denied the complaint's allegations. This is a case-specific enforcement action, not a ban on every identity tool. It illustrates why deployment purpose, review and routes to correct errors matter beyond matching accuracy.
9. Audit what the system passes as well as what it flags
A review queue reveals false alarms and ambiguous cases. It does not reveal defects the system confidently passed. Design an independent audit sample across decision bands, with known selection probabilities. Where feasible, assess items without first showing the model's verdict, then adjudicate disagreements using the appropriate reference method.
Keep the denominator and the sampling design visible
Estimate missed defects among passes and combine that evidence with verified defects in other bands to estimate overall recall. Use the sampling design and appropriate weights when selection rates differ. The proportion of defects among reviewed items is neither population prevalence nor recall. Returned products can reveal additional misses, but returns alone are a delayed and selected sample.
Rare defects require deliberate evidence planning. An enriched challenge set tests known difficult classes, while representative audits estimate field behavior. Report these separately. One hundred consecutive video frames of the same defect do not provide one hundred independent examples.
For illustration, observing zero misses among 300 independent, representative examples that truly have a particular defect gives a one-sided 95% binomial upper bound of about 1% on the miss probability. It does not prove a zero miss rate. This calculation assumes a stable process and correct reference labels; auditing 300 ordinary passed items answers a different question. Choose the study design and sample size for the consequence and required uncertainty.
10. Evaluate operational errors and task completion
Fix the evaluation protocol before tuning the final model and policy. Define each error, the reference standard, the matching rules for locations or events and the relevant conditions. Report uncertainty and sample counts, especially for costly classes and small groups.
Swipe horizontally to read all columns.
| Layer | Evidence | Release question |
|---|---|---|
| Capture | Usable-image rate and failed quality checks by condition | Does the camera reliably capture the required evidence? |
| Perception | Per-class recall, false alarms and localization errors | Are costly misses controlled at the chosen operating point? |
| Policy | Pass, referral and reject outcomes with audited labels | Are confident errors and review workload acceptable? |
| Execution | End-to-end tail latency and correct-item action checks | Does the intended action happen before its deadline? |
| Customer flow | Completion, corrections, retakes and handoffs | Does the experience improve the existing journey? |
| Resilience | Dirty lens, network outage, overload and rollback tests | Does the fallback work under realistic pressure? |
Detection and segmentation metrics help compare models, but their averages can hide the defects that matter. A tiny missing seal may be more costly than a large cosmetic mark. For video, evaluate event detection and duplicate alerts, not only individual-frame accuracy.
Model scores should not be presented as calibrated probabilities without evidence. Test error rates within the automated and review bands, by station and condition. A review band based on score thresholds does not reliably capture every uncertain case, particularly unfamiliar inputs. The risk scoring chapter explains the distinction between ranking, calibration and policy.
11. Release by station and maintain the capture system
Begin in shadow mode while the existing process remains responsible for decisions. Compare the candidate with independently established outcomes, not only inspector agreement. Introduce limited use with review and explicit boundaries, then expand only after the evidence supports the next population, station or action.
Version the whole operating package
Bundle model, preprocessing, taxonomy, thresholds, camera calibration and runtime compatibility. Verify artifact integrity and device compatibility before rollout. Test a candidate beside the current version only if the additional computation does not compromise production deadlines. Stage releases, retain a compatible rollback package and verify recovery on actual hardware.
Assign ownership for lens cleaning, lighting, mounting, recalibration and camera replacement. A moved camera or new supplier texture can change the input distribution without any software release. Monitor health and distributions alongside audited outcomes; embedding drift is a diagnostic signal, not proof that quality has deteriorated.
Active learning can prioritize uncertain cases, disagreement, diversity and newly observed conditions for labeling. Do not automatically treat operator feedback as ground truth or append every correction to training. Verify labels, preserve their provenance and keep an independent evaluation stream. Confident errors need audits precisely because uncertainty sampling may never select them.
12. Include review, downtime and false rejects in the economics
Evaluate incremental value against the current process: avoided defect costs, realizable labor savings and improved customer completion, minus additional scrap, rework, hardware, compute, labeling, maintenance and review. Avoid counting every detected defect as an avoided customer loss when an existing downstream inspection would already have caught it.
Calculate the human workload before scaling
As an illustrative capacity calculation, 10,000 items per shift with a 5% referral rate create 500 reviews. At 20 seconds each, that is about 2.8 hours of direct review time, before breaks, adjudication or handling. These are planning assumptions, not measured results. Arrival peaks and a slow physical inspection can require more capacity than the average suggests.
Measure how thresholds affect workload and both error types. Reducing referrals by accepting more cases can hide risk; sending everything to a person can erase the intended benefit. Test queue overflow and recovery. Fallback capacity must exist when a camera fails, not only during a quiet demonstration.
For customer photos, include upload, model calls, storage, retakes, assisted support and abandonment. Compare the full flow with the existing journey under a credible pilot design. Keep the model effect separate from changes to staffing, lighting or forms where the business needs to understand which investment created the improvement.
13. Deliver one complete visual workflow
A strong first delivery includes a decision contract, an optics and capture trial, a versioned taxonomy, representative data, a baseline model and a tested route for unsupported input. It also includes the review experience, audit plan, operating metrics, deployment package and accountable owners.
Progress through evidence gates rather than a fixed calendar: prove capture, establish reference labels, validate the proposed policy, demonstrate real workflow performance and test recovery. Hardware availability, access to the line and the rarity of important defects determine how long those gates take.
For an implementation discussion, bring examples of the images users produce today, the decision they need to make, the cost of mistakes and the available fallback. Those details reveal whether the best first investment is a model, better capture, clearer standards or a simpler product interaction.
Next in the series: scenario analysis and optimization—how to turn forecasts, constraints and business objectives into decisions that teams can inspect and act on.
Sources and further reading
This article develops chapter 10 of the AI Product Playbook. Its architecture, audit design and capacity calculations are implementation guidance and explicit illustrations, not measured Infinity Technologies client outcomes. Case examples retain their original scope; legal requirements need assessment for the intended use and jurisdiction.
- Beede and colleagues, A Human-Centered Evaluation of a Deep Learning System Deployed in Clinics for the Detection of Diabetic Retinopathy, CHI 2020.
- John Deere, See & Spray Select, for the camera-to-nozzle workflow in fallow fields.
- Roth and colleagues, Towards Total Recall in Industrial Anomaly Detection, CVPR 2022.
- NIST, Face Recognition Technology Evaluation: 1:1 Verification, including demographic and image-quality considerations.
- Federal Trade Commission, Rite Aid case record and 2024 stipulated order.
- Information Commissioner's Office, Key data protection concepts for biometric recognition.
- European Commission, Navigating the AI Act, including prohibited practices and their exceptions.
