Keep evaluating an AI system after it ships
Use a repeatable evaluation and review process as the model, data and users change.
Originally posted · Published here · 1 source · Resume reading
Shipping an AI feature should start a feedback process. Our social post made that general point; NIST's AI Risk Management Framework provides a concrete way to think about the work [1].
Give the process an owner
The framework groups activities into Govern, Map, Measure and Manage. These cover responsibility, context, evaluation and action. NIST describes them as activities across the system's lifecycle, rather than a fixed sequence that ends at deployment [1].
For example, consider a support assistant. Name the person who can stop it, define which questions it should answer, collect representative test cases and decide what happens when it gives an unsupported answer.
Revisit the evidence
When you change the model or the information it retrieves, run the same cases again. Add examples from real failures after removing sensitive information. Record which version produced each result so a comparison has meaning.
That workflow is an example of applying the framework, not a claim that a particular checklist appeared in the original carousel. The useful habit is keeping evaluation connected to decisions: a failed check needs an owner and a response, not just a number on a dashboard.
Keep this guide
Filed under AI agents
Sources
- NIST AI Risk Management Framework: core functionsairc.nist.gov
Next in this stack
Keyword LOOP
Five rules for building an AI agent that works
Keyword STARTER
Write skill metadata an agent can recognize
Keyword SIMPLEFIRST
Choose an AI workflow before building an agent