Assessing the risk of a use case before it starts

In one week, a team lead gets a proposal for an assistant that listens to the team’s meetings and writes the summary. The next day a second proposal arrives, for an assistant that drafts the internal newsletter from the updates people send in. On Thursday a third asks for an agent that reads the customer mailbox and answers the mail itself. Each proposal comes with a demo that went well, and each person asks for a yes by Friday.

The team’s AI policy says which tasks an agent may do. It doesn’t say which new use case is safe to start, and a demo doesn’t answer that question. In this lesson we answer it with a risk assessment: a short written method that rates each proposal before it starts and says which ones need a decision from someone with the authority to take the risk. You need a document or a chat assistant with no tools, and about fifteen minutes.

The method on one page

Write down the result of each step.

1. Name the domains the use case touches. Personal data, money, safety (the physical safety of people) and reputation cover most cases. Next to the domains, name the people who are affected when it goes wrong.
2. Rate the impact of the worst realistic mistake, and the likelihood that it happens, on scales the organization agreed in advance.
3. Combine the two ratings into a level.
4. Compare the level with the agreed threshold. Below it, the team goes ahead under its policy. At or above it, the use case needs extra controls, or a written acceptance from the person who answers for the risk.

Here are the scales this lesson uses. An organization may pick other words and other numbers. Everyone must rate on the same scales, so two ratings can be compared.

Impact of the worst realistic mistake
  1 minor        one person is inconvenienced, and it is fixed within a day
  2 moderate     a few people are affected, or a small cost, fixed within a week
  3 major        many people are affected, or personal data reaches people who
                 should not see it
  4 severe       customers or the public are harmed, a legal duty is broken,
                 or the loss can't be undone

Likelihood that it happens
  1 rare             not expected in a year of use
  2 possible         a few times a year
  3 likely           about once a month
  4 almost certain   every week

Level = impact x likelihood
  1 to 3 low    4 to 6 medium    8 to 9 high    12 to 16 very high

Threshold: 8. A level of 8 or more needs extra controls or a written acceptance.

The first step comes first on purpose. A rating made before the domains are named is a rating of the demo. The domains and the affected people are the blast radius of the use case: what a wrong step can reach, and who feels it. They decide how bad the worst realistic mistake is, and so they decide the impact rating.

How easily a mistake can be undone is part of its impact, which is why the impact scale says “fixed within a day” and “can’t be undone”. Sort the work by how easily it can be undone [1]. A sent email or a payment is final, and a draft nobody has read yet can still be thrown away.

The likelihood rating comes from the real volume of work and from how quickly the team would notice a mistake. A mistake caught in a draft never reaches a customer, and one that nobody notices for a month has a month to happen again.

Checkpoint · scenario

Where does the assessment start?

A hiring manager proposes an assistant that sorts incoming job applications into “interview” and “no”. You are asked to assess it before it starts. What do you write down first?

Rate the likelihood from the pilot results, since a pilot shows how often the tool gets it wrong.Write down the domains it touches and the people it affects: the applicants, their personal data, and the hiring managers who act on the sort.Ask the vendor for the accuracy figure, and use it for both the impact and the likelihood ratings.Start the tool on a few real applications, and rate it once the team has seen what it does in daily use.

DecideHintSkip

Scoring the three proposals

The team lead writes down the domains, the people affected and the two ratings for each proposal.

• Meeting summarizer. It runs in team meetings and in one-to-ones, where health and performance come up. The summaries are saved in the shared team folder. Domains: personal data, reputation. Impact 3, because personal data reaches people who should not see it. Likelihood 3, because the one-to-ones are weekly and the folder is shared by default.
• Newsletter drafter. An editor reads every draft before it goes out to the staff. Domain: reputation. Impact 2, likelihood 2.
• Customer mail agent. It reads the customer mailbox and sends its replies without a person reading them. Customers write about orders, refunds and their accounts. Domains: personal data, money, reputation. Impact 4, because a wrong reply can promise a refund or show one customer another customer’s details. Likelihood 3, because the mailbox gets thousands of messages a month.

The team lead can say yes alone only to a proposal below the threshold. A proposal at or above it needs extra controls, or a written acceptance from the person who answers for the risk. A small program applies the method to these ratings and prints the level of each proposal and whether it is at or above the threshold. Before you read its output, work out the three answers yourself.

Checkpoint · predict

Which proposals can't the team lead approve alone?

Which proposals need extra controls or the risk owner’s written acceptance before they start? The program prints one at or above threshold: line per proposal, in the order above, and a yes means the team lead can’t approve that proposal alone. Type the three values, yes or no, separated by spaces.

What is printed?

CheckHintSkip

The program’s output:

threshold: 8 (high)

use case: meeting summarizer
  domains: personal data, reputation
  affected: everyone in the meetings, and anyone discussed in a one-to-one
  impact: 3 (major)
  likelihood: 3 (likely)
  level: 9 (high)
  at or above threshold: yes

use case: newsletter drafter
  domains: reputation
  affected: the staff who read the newsletter
  impact: 2 (moderate)
  likelihood: 2 (possible)
  level: 4 (medium)
  at or above threshold: no

use case: customer mail agent
  domains: personal data, money, reputation
  affected: every customer who writes in, and the support team
  impact: 4 (severe)
  likelihood: 3 (likely)
  level: 12 (very high)
  at or above threshold: yes

The customer mail agent was the expected result. The meeting summarizer is the one that surprises people, because summarizing a meeting sounds harmless. The domains changed the answer. Once the list says “personal data” and “anyone discussed in a one-to-one”, the summarizer is a tool that copies health and performance remarks into a shared folder every week.

The newsletter drafter goes ahead under the team’s policy. The other two wait for the decision in the next section.

Pitfall: Rating likelihood from the demo

The customer mail agent’s demo answered twenty test messages, and every reply was right. The person proposing it rates the likelihood 1, rare. Rated that way, the level is 4 (medium), and the agent is below the threshold. The twenty messages were picked for the demo, and the real mailbox gets thousands a month, with refunds, complaints and messages in other languages among them. A demo shows that the tool can do the task. It doesn’t show how often it fails on the messages nobody picked. Rate likelihood from the volume and the variety of the real work, and from how quickly a mistake would be noticed.

At or above the threshold

A use case at or above the threshold doesn’t stop there. It gets one of two decisions, and both are written down.

The first is extra controls that bring the level down. The team lead changes the customer mail agent so that it drafts each reply and a person from the support team reads and sends it. The impact of a wrong reply is the same as before, but a wrong draft is now caught before a customer sees it. Rated again, the likelihood is 1 (rare) and the level is 4 (medium), below the threshold. Rare holds only while a person really reads each draft. The support team already read and answered every message by hand, so reading a draft is less work than they did before. If the drafts arrive faster than people can read them, the approvals stop being read, and the rating goes back up. The approval sits right before the step that can’t be undone, as the human in the loop section of the agent risk lesson describes. The rating is only as good as that control, so the record names it.

The second is acceptance in writing by a person with the authority to accept the risk. That is the person who answers for the domain when it goes wrong: for the customer mailbox, the head of customer service. The person who proposed the tool, or the team that wants it, can’t accept the risk on their own. A person with the authority may accept it for a limited time, or with a review date, and signs a record that says so. An organization may name a more senior person for the very high band, and the record then carries that person’s name.

For the meeting summarizer the team lead picks a control that shrinks the list of affected people. The assistant stays out of one-to-ones, and each summary goes only to the people who were in the meeting. The team lead rates it again with the smaller list, and the new ratings go in the record.

Whichever decision is made, the record holds the same lines.

Risk record: customer mail agent
Domains        personal data, money, reputation
Affected       every customer who writes in, and the support team
Ratings        impact 4 (severe), likelihood 3 (likely), level 12 (very high)
Decision       extra control: the agent drafts, a person reads and sends each reply
Rated again    impact 4 (severe), likelihood 1 (rare), level 4 (medium)
Decided by     head of customer service
Review         in three months, or after the first incident

The record matters later. When a wrong reply does reach a customer, the team can read what was assumed, who decided and which control was supposed to catch it, and fix the rating or the control. Without the record the same question gets a new answer from whoever is asked. An auditor, or the person who takes over the team next year, can review a decision only when it is written down. The EU AI Act lesson comes next, and it covers the uses that the law itself sorts into risk levels.

Checkpoint · choice

Who signs the acceptance?

The team decides not to add controls to the customer mail agent for now, and to ask for a written acceptance of its level instead. In this organization, the person who answers for an area signs the acceptances for it at every level. Who should sign?

The support team member who proposed the agent, since they know best how it works.The vendor of the agent, because the vendor built it and knows its failure rate.The head of customer service, who answers for the mailbox and its customers.The whole support team together, by a vote at the next team meeting.

CheckHintSkip

Exercise

Assess one AI use case from your own work, one that is proposed or that you already use, in a document or in a chat with an assistant that has no tools. Write down the domains it touches and the people a mistake would affect, rate the impact and the likelihood on the scales in this lesson, work out the level and compare it with the threshold. If the level is 8 or more, name one control that would bring it down, and the person who would have to accept it in writing if you added none.

A good result fits in a record like the one above, and every rating has a reason next to it that someone else could check. If a likelihood reason mentions a demo or a trial run, rate it again from the real volume of work. Which domain did you almost leave off the list?

Stretch: Show your assessment to a colleague and ask them to rate the impact and the likelihood without seeing your ratings. Where your numbers differ, write down which fact explains the gap.

More practice

Extra checkpoints on the same ideas, if you want them. You can finish the lesson without them.

Checkpoint · scenario

Six months later

Six months after an agent started, a wrong reply reaches a customer. The team looks for the assessment, and finds only a line in the meeting notes: “agent approved”. Nobody can say whether the rating was wrong or a control was missing. Which change lets the team answer that question after the next incident?

Keep writing one line in the meeting notes each time a tool is approved, as the team does now.After each incident, rate the use case again from scratch, using what the team knows then.Stop approving agents until the team can prove that the assessment method works.When each use case starts, write a record with the ratings, the decision, the control, and who decided.

DecideHint

Recap

1. A risk assessment starts with the domains a use case touches (personal data, money, safety and reputation) and the people a mistake affects. That list sets the impact rating.
2. Rate impact and likelihood on scales everyone agreed, multiply them into a level, and compare the level with the agreed threshold.
3. Sort the work by how easily a mistake can be undone [1], and put that in the impact rating. Rate likelihood from the real volume of work and from how fast a mistake is noticed. A demo shows the tool can do the task, and says little about how often it fails.
4. At or above the threshold, add a control that brings the level down, or get a written acceptance from the person who answers for the risk.
5. The record is what makes the decision reviewable. It holds the domains, the ratings, the decision, the control and the name of who decided.

You can now

• Sets policy, logging and escalation for agents in an organization
• Names what an agent action can reach and break

Mark lesson finishedI know this, skip it

Habit

The next time someone proposes a new AI tool for your team, ask which domains it touches and who a mistake would affect before anyone talks about how well the demo went.

References

1. Addy Osmani, Ivar Soares Urdalen, Leo Simons. Guardrails and safety: why agent safety differs, defense layers, injection, human in the loop. Agent Engineer Course. Course. AEC-10