AI Product Modernization Playbook · Article 02 of 12
A customer asks which policy applies to their account. The answer exists, but it is scattered across a help article, a contract amendment and a recently updated internal procedure. A useful AI knowledge feature must find the right evidence, respect access rights and explain its answer clearly enough to check.
Retrieval-augmented generation, or RAG, gives a model relevant material at answer time instead of relying only on what it learned during training. That makes it a practical foundation for product help, enterprise knowledge, document comparison and guided research. It also creates a product responsibility: deciding which sources the system may use, when they are valid and what to do when they disagree.
This chapter of Infinity Technologies’ AI Product Modernization Playbook turns that responsibility into a buildable workflow. The goal is a faster path to a correct, current, permitted and verifiable answer. The examples and rollout targets below are illustrative design choices, not reported Infinity client outcomes.
On this page
1. Start with the question your product must resolve
“Chat with your data” is too broad to be a useful first scope. Choose a recurring question with a known audience, an authoritative source set and a measurable consequence. Examples include explaining an account policy, finding an approved implementation procedure or comparing provisions across documents that the user can already access.
Write an answer contract before choosing a model: who is asking, what evidence counts, which date and jurisdiction matter, what the response must contain and where uncertainty goes. A policy assistant might return a short explanation, applicable conditions, a source passage and an effective date. A research assistant might produce a draft comparison that a person reviews.
Define the boundary between knowledge and action
A knowledge feature can explain how to request a refund without issuing one. It can identify a documentation gap without rewriting the policy. Keep source updates under an owner’s approval, and place transactional actions in a separately controlled workflow. The first article in this series covers the additional controls needed when an assistant can act.
Measure the existing experience first: time to a verified answer, repeat searches, escalations caused by missing information and the proportion of questions that have an approved answer at all. A model cannot repair an absent policy by sounding confident.
2. Choose the right knowledge workflow
The same interface can conceal very different evidence requirements. Separate these workflows in the product design so that one retrieval strategy does not quietly become the default for every question.
| Workflow | Evidence to use | Response boundary |
|---|---|---|
| Product help and policy | Published, approved content for the relevant product version and region | Explain the rule and its exceptions; link to the applicable passage |
| Customer-owned documents | The current tenant’s files, within the user’s permissions | Answer within the selected collection; make missing coverage visible |
| Enterprise knowledge | Documents with source-native access rules, owners and lifecycle metadata | Respect changing permissions and distinguish authoritative guidance from discussion |
| Live business questions | Governed metrics or read-only queries plus definitions | Report the period, filters, units and data timestamp |
| Cross-document synthesis | A bounded, explicitly selected source set | Produce a reviewable comparison with evidence for each material finding |
Swipe the table to see all columns.
Start where source quality and ownership are strongest. A smaller collection with clear authority can be a better product than a broad index whose contents nobody maintains.
3. Build two pipelines: ingestion and answering
The ingestion pipeline turns changing source material into retrievable evidence. The answering pipeline selects and uses that evidence for a particular authenticated user. Treat them as separate services with separate failure signals.
Ingestion preserves meaning and provenance
Connectors need to detect additions, changes and deletions. Parsing should retain headings, tables, lists and source locations. Canonicalization identifies duplicates and versions; chunking creates retrievable units; embeddings and keyword indexes make those units searchable. Every step should retain enough provenance to reconstruct the passage behind an answer.
Answering operates inside a verified scope
Resolve the user’s tenant, identity and entitlements from trusted application context. Interpret follow-up questions, select the relevant collection and time scope, retrieve candidates, rerank them and assemble an evidence package. Only then should the model compose an answer. A release check verifies evidence, permissions and support before the response reaches the user.
Record document and chunk versions, the retrieval query, selected evidence, policy decisions and model configuration. Apply appropriate access and retention rules to these records: observability must not become a second uncontrolled copy of sensitive knowledge.
4. Make source authority and freshness explicit
A search result has a similarity score. A trustworthy source also has an owner, a status and a period of applicability. Store a stable document identifier, source URI, version, title, tenant, owner, authority class, classification, content hash, review date and ingestion time. Where relevant, include jurisdiction and business-effective dates.
Keep two clocks
Ingestion time tells you when the system processed a file. Effective time tells you when its contents apply. A policy indexed today might take effect next month; an archived contract can still be the correct evidence for a historical question. Make “current” the deliberate default and require an explicit date for historical answers.
Define source precedence with the content owner. An approved policy may outrank an informal discussion, but a newer file is not automatically more authoritative. When equally authoritative sources conflict, present the disagreement within the user’s permitted scope and route it to an owner. Do not let the model silently reconcile incompatible rules.
Treat deletions as a product event
Removing a document must trigger invalidation of its chunks, embeddings and answer caches. Retained audit material or evaluation snapshots need their own justified retention and access policy. Track content freshness and permission freshness separately; a successful nightly content import does not prove that an access revocation has taken effect.
5. Enforce access before evidence reaches the model
The model should never decide whether a user belongs to a tenant or may read a document. Apply authorization in the retrieval service using authenticated identity and the source’s permission semantics. Preserve group membership, inherited permissions, explicit denials and link-sharing rules where the source uses them.
Both keyword and vector retrieval must return only authorized candidates to downstream reranking and composition. Fetching a broad result set and asking the model to hide restricted passages is not an access-control design. Any intermediate service that processes sensitive candidates must itself be inside the authorized processing boundary.
Revocation includes caches and conversations
An answer cache must account for tenant, entitlement scope, policy version, source versions and the question’s time scope. Recheck authorization near response release. Follow-up questions must not retrieve revoked material from a previous conversation turn or cached answer. Previously displayed information cannot be recalled from a user’s memory, but the product can prevent new disclosure and restrict stored history.
Set an explicit synchronization limit for each source. If current access cannot be established for a sensitive collection, exclude that collection until it recovers. The user can receive a useful limited answer or a neutral explanation of unavailable coverage without learning the titles or existence of restricted documents.
6. Improve retrieval before enlarging the prompt
Chunk around meaning: a procedure, a policy clause, a troubleshooting step or a table with its headers. Preserve the parent section so that a precise match can be expanded with relevant context. A fixed token window that splits a condition from its exception can create an apparently grounded but incorrect answer.
Combine exact and semantic matching
Keyword search is useful for error codes, product names and identifiers; vector search helps with paraphrases. Combine their authorized result sets, merge rankings and rerank a bounded candidate pool. Decompose genuinely multi-part questions when needed, but evaluate whether extra retrieval steps improve completion enough to justify their latency.
Anthropic’s Contextual Retrieval describes adding a short chunk-specific explanation before embedding and keyword indexing. This is a useful technique to test when isolated chunks lose meaning. Generated context remains a retrieval aid: validate it against the source and do not treat it as independent evidence.
Test with real permission filters
Approximate nearest-neighbor search can behave differently under restrictive filters. The pgvector documentation explains that filtering after an approximate index scan can leave too few matches; iterative scans, suitable indexing or partitioning can help. Measure retrieval quality for small tenants and narrowly authorized users, not only for an administrator who can see everything.
Use a shadow index for an embedding migration, keep query and document embeddings compatible, and compare the same evaluation set before switching traffic. More context, a knowledge graph or a larger model should each earn its place through a measurable improvement.
7. Require evidence for each material claim
A clickable citation proves that a link exists. It does not prove that the linked passage supports the sentence beside it. Attach claims to source passages and retain version information. Check that citations belong to the retrieved evidence, remain permitted and apply to the question’s date and scope.
Use deterministic checks for identifiers, access and lifecycle state, and calibrated semantic checks for whether evidence supports a claim. A model-based judge can help triage answers, but it is another fallible component. Review samples and disagreement cases with domain experts. Two citations copied from the same underlying document do not constitute independent confirmation.
Protect the answer path from document instructions
Retrieved content is evidence, not authority to change the application’s rules. A source can contain malicious instructions as well as useful facts. OWASP’s prompt-injection guidance explains this risk. Keep composition separate from privileged tools, constrain the output, test hostile documents and verify evidence; an instruction in the system prompt alone is insufficient protection.
For sensitive workflows, complete checks before streaming substantive answer text. Showing a restricted or unsupported claim and then retracting it does not undo the disclosure. A retrieval progress indicator can keep the experience responsive while verification runs.
8. Route live numbers through governed data
“What does our cancellation policy say?” and “How many accounts cancelled last month?” need different evidence. The first may be answered from an approved policy. The second should use a governed metric or a validated read-only query with the correct tenant, period, filters and business definition.
For a mixed question, retrieve the definition and request the measured value separately. Return both with their provenance. Validate units, currency, timezone, denominator and aggregation level before composition. Do not let the model estimate a missing value from a few retrieved rows.
Numbers explicitly stated in an applicable approved document can be cited directly. A contract’s notice period does not require a database query merely because it is numeric. The distinction is between quoting authoritative evidence and computing a live business result.
An illustrative account-renewal answer might combine the customer’s permitted contract clause with a current renewal date from the account API. If either is unavailable or conflicts with the other, the response should identify the unresolved part rather than invent a complete explanation.
9. Design an answer users can verify
Show the answer’s scope: the selected workspace or collection, the relevant date and whether the result describes current or historical rules. Let the user inspect a cited passage in context, ideally at the version used to generate the answer. Do not make them search an entire document to check a single condition.
A useful abstention explains what is missing and offers an authorized next step: narrow the question, select a period, open a relevant source or contact the content owner. Avoid generic apologies and hidden confidence percentages that users cannot interpret.
Offer specific feedback reasons such as wrong, outdated, incomplete or inaccessible source. Connect that feedback to the document, query and answer versions. If the evidence is missing, send the issue to content operations; if retrieval missed available evidence, send it to engineering. These require different fixes.
Fallback modes should be visible. Authorized extractive snippets or source-native search can remain useful when generation is unavailable. Apply the same access rules to every fallback; recovery must not broaden the user’s permissions.
10. Evaluate the system layer by layer
Create a source-linked evaluation set from real question patterns, with expected evidence, allowed scope and acceptable answers. Include unanswerable questions, conflicting versions, ambiguous dates, restricted documents, revoked access and deleted sources. Refresh examples when the underlying knowledge changes.
| Layer | What to measure | What a failure suggests |
|---|---|---|
| Retrieval | Recall of required evidence at k; ranking quality; performance by tenant and permission scope | Parsing, chunking, filtering, query interpretation or ranking needs attention |
| Answer quality | Correctness, completeness, claim support and citation applicability | The evidence package or composition rules are inadequate |
| Abstention | Correct refusal on unsupported questions and unnecessary refusal on answerable ones | The release policy is too permissive or too restrictive |
| Access and lifecycle | Cross-tenant tests, revoked-cache tests, effective-date traps and deletion propagation | Authorization or freshness controls are failing |
| Product value | Time to verified resolution, repeat attempts, escalations and useful-answer cost | The feature may not improve the workflow despite acceptable model scores |
Swipe the table to see all columns.
An illustrative pilot could target 90% retrieval recall at 10 and 97% supported material claims on a reviewed benchmark. These are planning examples, not universal release standards. Define the denominator, sample size, risk slices and review method before adopting a threshold. Passing an aggregate score cannot compensate for a known access-control failure.
Calibrate automated judges against human reviews, especially after changing prompts, models or source types. Track over-abstention alongside unsupported answers; a system that refuses everything is safe from some errors but provides no useful knowledge service.
11. Learn from published deployments without copying their targets
Uber: measure the content as well as the answer
In its October 2024 engineering account, Uber described Genie, an on-call copilot that used curated engineering knowledge in Slack, offered human escalation and evaluated source documents as well as responses. Uber reported a 48.9% helpfulness rate at that point. Helpfulness is not a correctness score. Our takeaway is operational: knowledge quality and an escalation path belong in the product, even when retrieval and generation are already working.
LinkedIn: match retrieval to the support problem
The authors of Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answering reported a 28.6% reduction in median per-issue resolution time in a LinkedIn deployment described in their 2024 paper. That is a result from their system and setting, not a forecast for another product. The useful design lesson is to test whether relationships between issues, entities and evidence improve the chosen workflow before adding graph infrastructure.
Neither example removes the need for a local baseline. Your source coverage, access model, question mix and definition of success determine whether a similar approach creates value.
12. Roll out one source and one audience at a time
Begin with one high-value collection and a named content owner. Use a limited audience to compare the feature with the existing search or support path. Review failed answers together with source gaps, then expand only when quality and ownership remain manageable.
- Establish the contract. Agree on the audience, approved sources, response format, baseline and acceptance criteria.
- Prove the evidence path. Validate parsing, versioning, permissions, citations and useful abstention with a representative test set.
- Exercise failure and recovery. Revoke access, delete a source, introduce conflicting versions and simulate an unavailable connector. Test the fallback before users need it.
- Release a bounded pilot. Monitor real questions, sample answers for independent review and route documentation issues to owners.
- Expand deliberately. Add a collection, audience or retrieval technique only after measuring the effect on quality, latency and cost.
Calculate cost across ingestion, embedding refreshes, storage, retrieval, reranking, generation, evaluation and human review. Divide by independently assessed useful answers, not total messages or the absence of a downvote. Report uncertainty when usefulness is estimated from a sample.
Assign operational owners for connectors, access synchronization, content maintenance and evaluation. Define rollback as a product behavior: disable a problematic collection or generation route while retaining authorized search and a clear path to help.
13. Turn the Playbook into a concrete product increment
A practical discovery package for a knowledge feature should contain an answer contract, a source and ownership map, a permissions design, an evidence schema, an evaluation set and a rollout decision. These artifacts let product, engineering and domain owners review the same intended behavior.
Infinity Technologies’ Playbook approach starts with that bounded workflow and connects it to the existing product: authentication, source systems, the user interface, observability and support. The implementation question is specific: can a defined audience resolve a defined class of questions faster while keeping evidence current, permitted and checkable?
For related implementation perspectives, explore the HR Agent guide for employee-facing knowledge workflows and the InfinitySDLC platform foundation for shared platform concerns. To scope a knowledge feature for your product, connect with Infinity Technologies around one workflow and its current evidence sources.
Next in the series: document processing—turning files, scans and forms into validated product data.
Sources and further reading
This article is based on the Knowledge and RAG chapter of Infinity Technologies’ AI Product Playbook. External examples and technical references were checked on 13 October 2026. Diagrams are original explanatory illustrations; the HERO is an original AI-generated editorial image.
- Uber Engineering: Genie—Uber’s Gen AI On-Call Copilot (2024).
- Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answering (2024 research paper).
- Anthropic: Introducing Contextual Retrieval.
- pgvector: filtering, iterative scans and hybrid search.
- OWASP GenAI Security Project: prompt injection.
