Testing a skill from a fresh session

The last lesson ended with a release skill that works in the directory it was written in. This lesson runs it somewhere else: a clean clone, and a new session that has none of the conversation you had while you wrote it. Anthropic’s guide to writing skills describes the same loop. One session helps you write the skill, a fresh one uses it on a real task, and you watch where the fresh one struggles and bring that back into the file [1].

First you run the skill and read the transcript for the places where the agent hesitated, asked you something or guessed. Then you turn each of those places into a line in the skill, run it again and compare the two runs. Two short Python scripts show what the agent sees in each repository, and the course checks their output on every build.

Run it where it wasn’t written

A clean clone is a copy of the package with its own git repository and nothing else. It has none of the author’s tags, history or uncommitted files. Make one outside the course repository. Claude Code reads the CLAUDE.md files in the directory it starts in and in every directory above it [2], so a copy inside the course repository gets the course’s own agent instructions as well.

Auto memory saves what one session learned for the sessions after it. It is on by default, and it is kept per git repository [2]. The same page says to turn it off for one project by setting autoMemoryEnabled to false in that project’s settings, and the commands below do that in .claude/settings.local.json. They also tell git to ignore that file, so it never shows up as a change. Your personal instructions in ~/.claude/CLAUDE.md load in every session, in every project [2], and they still reach the test.

Run these commands from the top of your copy of the course repository, so that the cp line finds the package. The first line deletes ~/notes-clean, so the commands also work on a second attempt. The rm -rf __pycache__ line removes compiled files that are in the fixture directory if you ran its check in the course repository.

rm -rf ~/notes-clean
cp -R site/examples/customizing-agents/first-skill/fixture-package ~/notes-clean
cd ~/notes-clean
rm -rf __pycache__
git init -q
echo '{ "autoMemoryEnabled": false }' > .claude/settings.local.json
mkdir -p .git/info
echo .claude/settings.local.json >> .git/info/exclude
git add .
git commit -q -m "Import the notes package"

The copy contains the release skill from the last lesson at .claude/skills/release/SKILL.md. Start your agent in ~/notes-clean, in a new session, and type /release 0.4.0. In Claude Code the skill’s directory name is the command you type [3], and in another agent you can ask for release 0.4.0 in plain words. The skill changes files, commits and tags, and that’s why it runs in this copy and nowhere else. The copy has no remote, so a push has nowhere to go.

Step 3 of the skill says to write one changelog bullet per change since the previous release and to read git log for the list. The first script builds two repositories and asks each one what step 3 needs. The author’s repository has the package committed as release 0.3.0, a v0.3.0 tag on that commit, and one change after it. The clean clone is the one you just made.

Checkpoint · predict

What does step 3 have to work with?

Predict the output for both repositories. The script prints none where git describe --tags --abbrev=0 fails.

python3 fresh_clone.py

What is printed?

Output verified in CI from site/examples/customizing-agents/testing-a-skill/fresh_clone.py.

CheckHintSkip

In the author’s repository, step 3 has one commit to turn into one bullet. The clean clone gives it no tag to count from, and its only commit contains the whole package. The skill is silent about that case, and the agent has to choose for itself. Here is the kind of run that follows. It is written by the course to show the pattern, and it is not recorded from a model. Your agent’s words differ.

Prompt (illustrative, not a recorded transcript)

/release 0.4.0

Response

Tests pass: OK.
There are no tags, so I can't find the commit for 0.3.0. The log has one
commit, "Import the notes package". I'll describe the package as it is.
Updated notes.py to 0.4.0.
Added to CHANGELOG.md:
  ## 0.4.0
  - Notes command with list, add and --version.
release 0.4.0: ok
git status shows an untracked __pycache__/ directory. Should I include it
in the release commit?

Read the transcript as a list of places where the skill gave no instruction. The agent guessed at the change list and wrote a bullet for features that shipped in earlier releases. It asked about __pycache__/, which the test run in step 1 leaves behind. The check still printed ok, because it looks for a changelog heading and never reads the bullets under it. Each of those places is a line the skill is missing.

Turn each hesitation into a line

In this run the hesitations came from a skill that assumed something about the environment, such as a tag or a clean working tree. In other runs they come from a step that assumes a tool is installed, or from a success test too vague to tell a right result from a wrong one. Anthropic’s guide says the same about scripts: don’t assume a package is installed, and name what the skill needs [1].

The fix for the first two is a short list of checks at the top of the skill, each with what to do when it fails. Add this section above ## Steps:

## Before you start

Run these checks in order. If one fails, stop, and report the command, what
it printed and what the user has to do. Don't fix the repository yourself.

1. `git --version` and `python3 --version` each print a version. If one
   prints an error, that tool is missing. Name it and stop.
2. `git status --porcelain` prints nothing, or only `?? __pycache__/`.
   An error means this directory is not a git repository. Any other line
   is an uncommitted change. Say which, and stop.
3. `git describe --tags --abbrev=0` prints the tag of the previous
   release, `v` and the `__version__` in `notes.py`. If it fails, there is
   no release tag. Ask the user to tag the commit of the previous release,
   and stop. Don't work that commit out from the changelog.
4. `git log --format=%s <tag>..HEAD` prints at least one line. If it
   prints nothing, there is nothing to release. Say so and stop.

Then point step 3 at that log, and make step 5 say what to commit:

3. In `CHANGELOG.md`, add a `## <version>` heading under the title, above
   the previous release, with one bullet per user-visible change in the
   log from check 4. Leave out a commit a user would not notice. Never
   write a bullet that no commit in that log supports.
...
5. Stage `notes.py` and `CHANGELOG.md` only, and commit with the message
   `release: <version>`. Step 1 leaves a `__pycache__` directory behind.
   Don't commit it. Then tag the commit with `git tag v<version>`.

Check 3 is the one that answers the transcript. The agent guessed because the skill gave it nothing to do when the tag was missing. Now it has an instruction, and the instruction is to stop. A skill that stops with a clear question is better than one that finishes with an invented changelog, because you see the stop, and you might not read the changelog.

Checkpoint · match

Which line answers which hesitation?

Match each place where the agent hesitated to the line that answers it.

The agent found no tags and wrote bullets for features from earlier releasesChoose…Check that python3 --version prints a version, and stop if notStop and ask when git describe --tags --abbrev=0 failsStage notes.py and CHANGELOG.md onlyDone when git describe --tags prints exactly the new tag

The agent asked whether the untracked `__pycache__/` goes in the release commitChoose…Check that python3 --version prints a version, and stop if notStop and ask when git describe --tags --abbrev=0 failsStage notes.py and CHANGELOG.md onlyDone when git describe --tags prints exactly the new tag

On a colleague's laptop the first command failed with "command not found"Choose…Check that python3 --version prints a version, and stop if notStop and ask when git describe --tags --abbrev=0 failsStage notes.py and CHANGELOG.md onlyDone when git describe --tags prints exactly the new tag

The agent reported the tag as done because `git tag --list` showed itChoose…Check that python3 --version prints a version, and stop if notStop and ask when git describe --tags --abbrev=0 failsStage notes.py and CHANGELOG.md onlyDone when git describe --tags prints exactly the new tag

CheckHintSkip

The last row is the vague success test. The first version of the skill’s “Done when” section says git tag --list 'v*' shows the new tag. That is true whenever the tag exists, wherever it points. The second fixture releases 0.4.0 twice in the author’s repository, once with git tag run before the release commit and once after it, and prints three checks each time.

Checkpoint · predict

Which check tells the two releases apart?

Predict the six lines under the two headings. The script replaces the commit hash that git appends with <hash>.

python3 tag_placement.py

What is printed?

Output verified in CI from site/examples/customizing-agents/testing-a-skill/tag_placement.py.

CheckHintSkip

Only git describe --tags differs. When the tag is one commit early, v0.4.0 points at the commit before the version bump, and anyone who checks out that tag gets 0.3.0 code. So the new “Done when” says:

`python3 release_check.py` prints `release <version>: ok`,
`git describe --tags` prints exactly `v<version>`, and
`git status --porcelain` prints nothing, or only `?? __pycache__/`.
Don't push.
Report the version, the tag and the changelog entry, with the commit
each bullet came from.

The whole revised skill is in the course repository at site/examples/customizing-agents/testing-a-skill/release/SKILL.md.

Run it again and compare

First put the clone back as it was before the first run. These two commands delete the v0.4.0 tag, and move the branch back to the import commit, which drops every change after it. git tag -d prints an error if the first run made no tag, and that is fine. The reset also drops edits you haven’t committed, so run it before you revise the skill.

cd ~/notes-clean
git tag -d v0.4.0
git reset -q --hard "$(git rev-list --max-parents=0 HEAD)"

Now write your revisions into .claude/skills/release/SKILL.md, or copy the revised file from the course repository, and commit the change. The new check 2 stops on any uncommitted change, and that includes an edited skill. In the cp line, replace <course-repo> with the path to your copy of the course repository. Leave the cp line out if you wrote your own revision.

cd ~/notes-clean
cp <course-repo>/site/examples/customizing-agents/testing-a-skill/release/SKILL.md .claude/skills/release/SKILL.md
git commit -q -a -m "Revise the release skill"

Claude Code picks up an edited skill in the session that is already running [3], but start a new session anyway. The old session still holds your answer about __pycache__/, and a rerun in it tests the skill plus that answer. Auto memory is off in this clone, and in Claude Code you can run /memory to see which memory files a session can load [2]. Type /release 0.4.0 again.

A good run now stops at check 3. The agent reports that git describe found no tag and asks you to tag the commit of the previous release. That is the skill working: the repository lacks a fact that no agent can work out, and the skill says to ask for it.

Checkpoint · choice

What does the rerun tell you?

In the rerun, the agent stops at the new check 3 and asks you to tag the commit of the previous release. What does that result tell you about the revised skill?

The skill has failed, since a good skill completes the release without asking the user anything.The skill needs one more line that takes the previous release from the newest changelog heading.The skill has worked: the repository lacked a fact, and the agent stopped and asked instead of guessing.The rerun tells you nothing yet, since the agent stopped before it reached any of the release steps.

CheckHintSkip

Now give the clone the history the skill expects. Tag the import commit, which holds release 0.3.0, and make one change after it:

cd ~/notes-clean
git tag v0.3.0 HEAD~1
echo 'Notes are stored in `notes.json`, next to `notes.py`.' >> README.md
git commit -q -a -m "Document where notes are stored"

Start one more new session and type /release 0.4.0. Compare the transcript with the first one. Check 4 lists two commits, the skill revision and the README change. A user never sees the skill, so a good run writes one bullet, for the README change, and reports the commit it came from. After it, git describe --tags prints v0.4.0. If the agent hesitated somewhere new, that is the next line to add, and the loop is the same: run, read, revise, run again.

Pitfall: The skill that only worked at home

An engineer writes a release skill in their own clone, runs it three times there, and shares it. On a colleague’s laptop the agent writes a changelog from the whole history and commits a __pycache__ directory. The engineer’s clone had the tags, and weeks earlier they had told their agent not to commit __pycache__. Claude Code can keep a correction like that in auto memory, which is on by default and stored per repository on one machine [2]. The skill never said it, and the engineer’s agent remembered it for them. Test a skill in a clean clone in a new directory, and ask a colleague to run it once on their machine before you rely on it.

Exercise

If you followed along with the walkthrough, your runs in ~/notes-clean are the exercise, and you can go to the result below. Otherwise make a clean clone with the commands in the first section. They delete and recreate ~/notes-clean, and the clone has the release skill from the last lesson in it. The skill edits notes.py and CHANGELOG.md, commits and tags, so run it only in that copy, which has no remote to push to. In a new session, type /release 0.4.0, and write down each place in the transcript where the agent hesitated, asked you something or guessed. For each one, add a check, a failure step or a sharper “Done when” line to the skill. Then reset the clone, commit the revised skill and rerun in a new session, as in the last section.

A good result: your list has at least two places, each with the line that answers it. The rerun stops at the missing tag instead of guessing, and after you tag v0.3.0 and make one change, it finishes with one changelog bullet and git describe --tags printing v0.4.0. Which of your hesitations came from something you knew and never wrote down?

Stretch: Ask a colleague to run your revised skill in their own clean clone, on their own machine, without help from you. Add one line for each question their agent asked.

Recap

1. Test a skill in a clean clone and a new session, with auto memory off [2], so the conversation you had while writing it can’t fill a gap for the agent [1]. Your personal instructions still load, and a colleague’s machine is the test without them.
2. Read the transcript for each place the agent hesitated, asked or guessed. Each one is a line missing from the skill.
3. A missing line is often a check on the environment before the steps, a step that names the tool or state it needs, or a success test exact enough to fail a wrong result.
4. When a fact is missing from the repository, the right line tells the agent to stop and ask. A clear stop is better than a guess that passes the check.
5. Rerun in a new session after each revision, and compare the two runs.

You can now

• Packages a repeatable procedure as a skill another person's agent can use

Mark lesson finishedI know this, skip it

Habit

The next time you write or change a skill, run it once in a new session on a clean copy before you share it, and read where the agent hesitated.

References

1. Anthropic. Skill authoring best practices. Claude Platform documentation. Reference. Claude docs best-practices
2. Anthropic. How Claude remembers your project. Claude Code documentation. Reference. Claude Code memory
3. Anthropic. Extend Claude with skills. Claude Code documentation. Reference. Claude Code skills