Four source records validated; role filtering excluded the admin-only record; all 3 authorized sources reconciled to indexed chunks with SHA-256 lineage.
PyTorch / Transformers / pytest / CPU inference
Small-Model AI QA Proof of Concept
An executed qualification harness for RAG, data lineage, nondeterministic generation, and agent tool policy using the 135M-parameter SmolLM2-Instruct model as the system under test.
What this proves
A red build can be the most credible result.
The PoC pins an Apache-2.0 model revision, runs locally on CPU, and retains every prompt, seed, raw generation, latency, and gate decision. The data and retrieval layers passed. Generation did not reliably obey grounding, abstention, citation, or tool-call contracts, so the release was correctly held rather than softened into a favorable demo.
- System under test
- HuggingFaceTB/SmolLM2-135M-Instruct
- Execution
- CPU / PyTorch 2.10 / Transformers 5.3
- Harness tests
- 13/13 deterministic pytest checks passed
- Safety boundary
- 0 model-generated side effects executed
- Decision
- Release held by measurable quality gates
September 24, 2026 execution
One run, four independently observable layers.
Passing infrastructure is not confused with passing model behavior. Each metric answers a different failure-localization question.
The relevant source ranked into context for three answerable cases, while the unsupported roof-color query correctly retrieved no source after threshold hardening.
The model answered one fact correctly but cited an example ID, omitted required citations elsewhere, and invented a roof color despite an empty evidence set.
Ten seeded trials produced ten unique outputs and only five complete passes, exposing output variance hidden by a single deterministic run.
Every generation violated the JSON tool-call contract. The parser and policy layer rejected all four, including the destructive request, before execution.
Independent tests cover source contracts, duplicate IDs, ACL filtering, reconciliation, retrieval confidence, citation validity, abstention, schemas, and authorization.
Executed architecture
A small model inside hard, inspectable boundaries.
Contract and lineage
Validate IDs, required fields, timestamps, ACLs, duplicate records, hashes, and source-to-index counts before inference.
Rank and authorize
Use a transparent token-overlap retriever, role-filtered chunks, confidence threshold, expected source IDs, and absent-evidence fixtures.
Repeat and score
Pin model and prompt inputs, retain raw output, repeat across seeds, then score facts, abstention, citations, latency, and variance separately.
Parse before effect
Require exact JSON, allowlisted tools, typed arguments, expected authorization, and local fake tools. Invalid output fails closed.
Failure localization: retrieval passed while generation failed. That evidence points remediation toward model capability, constrained decoding, and refusal behavior rather than corpus ingestion or ranking.
Findings and next experiment
The PoC establishes a baseline, not a production claim.
- Prompt hardening raised retrieval qualification from 3/4 to 4/4 by preventing a generic one-token match from entering context.
- The model copied a demonstration citation ID instead of the retrieved source ID, proving citation presence alone is an inadequate oracle.
- With no retrieved evidence, the model invented a roof color instead of returning the required abstention token.
- Sampled pass rate was 50% across ten fixed seeds, below the 80% release threshold and too variable for the proposed contract.
- All agent responses failed schema validation; fail-closed orchestration prevented model prose or code from becoming an action.
- The next comparison should hold fixtures constant while adding grammar-constrained decoding or testing a slightly larger model.
Reproducible evidence
Inspect the run, not a screenshot of a claim.
Model weights are intentionally excluded. The source archive downloads the pinned revision on first execution and produces the same report shape locally.
Evaluation JSONPrompts, outputs, seeds, latency, scores, and release decision
JUnit13 deterministic harness checks
Initial BaselinePre-hardening retrieval and generation evidence
Qualification SummaryHuman-readable scope, results, and disposition
Runnable SourcePython harness, fixtures, tests, and pinned model revision
Model CardUpstream Apache-2.0 model documentation