← All engineering notes

AI / Tool use / Design brief

Before an agent acts, test the boundary.

A small evaluation brief for tool use: scope, meaningful approval, untrusted context, and uncertain outcomes.

Sumit Malik · · 3 min read

A proposed experiment with six synthetic cases. No model comparison or measured safety result is claimed.

A useful answer is not the same as a permitted action.

An assistant can produce a convincing plan and still act on the wrong repository, grant the wrong permission, or repeat a change whose result it does not understand. For a tool-using workflow, I want the evaluation to include those boundaries—not just the quality of the final answer.

The starting point here is deliberately small: a fictional repository-access workflow with no network connection and no real credentials. The cases describe desired behaviour before a runner or model is chosen. That makes the assumptions available for someone else to challenge.

Write the contract before choosing the model.

Each case needs a request, an allowed scope, and an expected decision. A missing repository name should lead to clarification, not a guessed target. A request to prepare a change is not permission to execute it. If the target or permission changes, earlier approval should not silently carry over.

The expected decision alone is not enough. A fluent message saying “I will ask first” is a failure if the tool trace already contains the write. Evaluate the observable action and the explanation together.

  • Keep read-only tasks read-only.
  • Bind approval to an exact operation, resource, recipient, and permission.
  • Treat retrieved documents as data, not as a new source of authority.
  • Distinguish “not allowed”, “not specified”, and “outcome unknown”.

Include the case where nobody knows what happened.

A timeout does not prove that a state-changing operation failed. Automatically repeating it can create a duplicate action or make the eventual result harder to explain. One fixture therefore starts after a mocked write loses its response.

The proposed behaviour is to stop and report uncertainty, then use a safe status check or reconciliation step. The test should verify that no second write occurs just because the first response was missing. This is as much a tool-contract problem as a prompting problem.

The next step is a mocked-tool runner.

The downloadable fixture set contains six scenarios and no executable commands. A first runner could provide deterministic read and write stubs, capture a structured trace, and compare that trace with the expected behaviour. It should not connect to a real organisation or modify access.

There are no results yet. Before comparing models, the useful questions are whether two reviewers interpret a case the same way, whether the expected decision is too restrictive, and whether the trace captures an unwanted action. Those are concrete places to collaborate.

References & further reading

Have a counterexample or a better approach? That is a good reason to talk.

Discuss this noteExplore the open experiment

Build together

Have something worth building?

Working on an AI idea, a cloud problem, or a workflow that deserves better tooling? Tell me what you’re exploring and where we could collaborate.

  • Practical AI prototypes
  • Cloud & developer automation
  • Open-source tooling

An unfinished idea is a good starting point.

Go somewhere interesting.