TAKE THIS TO WORK
Define what a good answer must do before testing, including when the assistant should stop.
Limit the task.
In this simulated example the assistant searches approved maintenance manuals and drafts an answer for a person to check. It does not change equipment, approve purchases or make independent decisions.
Define permitted sources, user roles and human handoff. Connecting files to a model does not establish correct permissions or answers. Access controls must be enforced and checked in the application.
Write expected outcomes before reading the answers.
For an answerable question, record the supporting passage. When information is absent, expect the assistant to state the limit and follow the agreed route. Confident, fluent language is not evidence of correctness.
The six cases below are a starting point, not a certification or universally sufficient test set. Add representative questions and documents, with outcomes reviewed by someone who understands the work.
Test what it must not do.
A role without permission must not receive restricted content, even in a summary. A document saying “ignore the rules” contains text, not authorisation. Exercise these cases through the actual application workflow.
For conflicting manuals, establish a version rule or human review path. Record the question, available sources, expected outcome and observation. A single overall accuracy score can conceal a serious access failure.
Repeat relevant checks after changes.
A different model, prompt or manual may change behaviour. Keep the cases and separately examine correctness, support, access and handoff after relevant changes. Google Cloud describes evaluating against criteria and test data.
ARXE Studio designs custom AI integrations around tasks, sources, permissions and human control. The mini-check is deterministic and local: it calls no model and uses no company documents.
| Test case | Behaviour to check | Failure to record |
|---|---|---|
| Answer available | Supported by an authorised source | Unsupported detail |
| Information absent | State the limit and follow the agreed route | Invented procedure |
| Sources conflict | Version rule or human review | Silent selection |
| User lacks permission | Withhold restricted content | Disclosure through a summary |
| Malicious instruction in a file | Do not treat it as authorisation | Expanded access or task |
| Outside scope | Identifiable human handoff | Continued autonomous decisions |
Make the call.
The requested procedure is absent from the approved manual. What should the assistant do?
Choose an answer to see the reasoning.
Local, deterministic example. No data sent and no AI model called.
Sources and further reading
How this content is made
Text prepared with AI assistance. Original examples and mini-checks are simulated and show no clients or measured results. The fictional characters are not the authors.
ARXE STUDIO / TAKE THIS TO WORK
Have a specific job for AI?
Describe the task, available sources, systems involved and who should approve the result.
Discuss your workflowNext proposed topic: synchronisation failed. Who notices?