Trying two models on the same task

In this lesson we give the same three small tasks to two models and watch what each one does with them. Most chat tools have a model picker with at least two entries, a small one that answers fast and a large one that answers slower and costs more per message. Many people leave the picker on the largest model for everything, because “the best” sounds like the safe choice. By the end of the lesson you have a six-cell table for these three tasks. It shows which task the large model’s extra price and delay paid for, and you have a way to build the same table for a task of your own.

You need a chat tool whose model picker offers a small and a large model. Any pair works, and you don’t have to know the names. Pick the smallest and the largest, and call them “small” and “large” in your notes. The answers shown on this page are illustrative. The author wrote them to show the pattern, and your models give a different text.

Three tasks, one prediction each

The inputs follow. Read each one and, before you run anything, write down which of the two models you think it needs. That prediction is the point of the section. Once you have run the tasks you can’t recover what you would have guessed.

The first is reformatting. The text goes in, the same facts come out as a table, and nothing has to be decided.

Prompt (illustrative, not a recorded transcript)

Turn these lines, in the order given, into a table with the columns Room, Floor, Seats, and Screen.

Harbor room, second floor, seats 8, has a screen Meadow room, ground floor, seats 4, no screen Summit room, third floor, seats 12, has a screen Quarry room, ground floor, seats 6, no screen

The second is classification. Each message gets one label from a fixed list, and the list is short enough that the answer is either right or wrong.

Prompt (illustrative, not a recorded transcript)

Label each of these customer messages as billing, delivery, or other. Reply with one line per message: the number and the label, nothing else.

1. I was charged twice for the same order last week.
2. The parcel says delivered but nothing has arrived.
3. Can I change the email address on my account?
4. The invoice shows the old company name.
5. Your driver left the box in the rain.

The third is an ambiguous request. The useful reply depends on facts the message doesn’t give, and a good answer either asks for them or lays out how the advice changes with each one.

Prompt (illustrative, not a recorded transcript)

Our team offsite is in six weeks and the budget was just cut by a third. What should we do?

Write your three predictions down now. A common guess is “large for all three, to be safe”. Hold on to yours and see how it does.

Checkpoint · match

Name the work before the model

Match each request to what it asks the model to do.

Turn these four lines about rooms into a table with four columns.Choose…ReformattingClassificationMulti-step reasoningFresh facts

Label each of these five messages as billing, delivery or other.Choose…ReformattingClassificationMulti-step reasoningFresh facts

Our offsite budget was cut by a third. What should we do?Choose…ReformattingClassificationMulti-step reasoningFresh facts

What did our main competitor announce this week?Choose…ReformattingClassificationMulti-step reasoningFresh facts

CheckHintSkip

Run them on both models

Now run all three inputs on the small model, then on the large one. Use a fresh chat for each run, so no earlier answer is in the context. For each of the six answers, note two things: how long it took to finish, counted on a clock or by feel, and whether the answer is good enough for the task. “Good enough” means you would use it as it is. Write the six verdicts in a table with a row per task and a column per model.

Here is what one run looked like for the author, with the small model’s answers first. This is one example run, written to show the pattern. It is not a measurement of any model, and your table may differ.

Response

| Room   | Floor  | Seats | Screen |
|--------|--------|-------|--------|
| Harbor | Second | 8     | Yes    |
| Meadow | Ground | 4     | No     |
| Summit | Third  | 12    | Yes    |
| Quarry | Ground | 6     | No     |

The small model reformats in a second or two, and every cell matches the input. The large model gives the same table a few seconds later. The task didn’t need the extra time.

Response

1. billing
2. delivery
3. other
4. billing
5. delivery

The same on classification. Both models label the five messages the same way, and the small one is done first. The one place you might see a difference is message 4, where “invoice” and “company name” pull toward two labels. Both models in the example run chose billing, which is the label the fixed list makes most useful.

Response

Here are some ways to handle a budget cut for a team offsite:

1. Shorten it. Two days instead of three cuts hotel and meal costs.
2. Move it closer. Travel is often the largest line item.
3. Go remote for part of it. Run the workshops online and meet in
   person for one day.
4. Cut the extras. Drop the outside speaker or the evening activity.
5. Ask for sponsorship. Some vendors will co-fund an event.

Would you like me to expand on any of these?

The small model’s answer to the ambiguous request is a generic list. Every item could apply to any offsite anywhere. The one fact in the message, that a third of the budget is gone, appears in none of them. The answer is fluent and beside the point. As you saw in the lesson on how models work, the model produced the most likely text for a question like this, one token at a time, and nothing in that process notices what the question left out.

Response

Before choosing, it helps to know what the missing third was going to
pay for, because that decides which options are open.

- If the cut lands on travel, the offsite can move to a location most
  of the team can reach by train, or go hybrid with one in-person day.
- If it lands on venue and catering, a shorter event keeps the agenda
  and drops one night's hotel and two meals.
- If it lands on the program, the outside facilitator goes and a team
  member runs the sessions, which changes the tone more than the
  content.

Two questions would narrow it down: what was the original budget split,
and which parts of the offsite are the reason it exists at all? Protect
those parts and cut around them.

The large model’s answer takes longer to arrive and is the one you could act on. It names the missing fact, shows how the choice depends on it, and asks the two questions whose answers pick the option. This matches what the concept of model fit describes: a small model handles reformatting and classification, and a larger one handles reasoning about an ambiguous request [1]. A run like this one is how you find out the fit. The size of the model on its own tells you nothing about it.

The example run’s table then reads like this. “Yes” means good enough to use as it is.

Task	Small model	Large model
Reformatting	Yes, fast	Yes, slower
Classification	Yes, fast	Yes, slower
Ambiguous request	No	Yes

Checkpoint · choice

Where the large model earns its price

Of the three tasks in the run, which one justifies the large model, and why?

The reformatting, because a large model is more careful with the cells of a table.The classification, because a large model knows the difference between billing and delivery.The ambiguous request, because the useful answer needed a fact the message left out.All three, because a large model is more accurate on every task and speed does not matter in a chat.

CheckHintSkip

Read the table at volume

For one chat, the difference between the two columns is a few seconds and a price too small to notice on a monthly bill. That is why the picker set to “large” feels free. The table starts to matter when the task repeats.

Suppose the classification task is real: your team gets five thousand customer messages a day, and someone wants a model to label each one. From your run, the small model labeled the five messages as well as the large one. Now scale the other two columns. If the large model took four seconds per answer where the small one took one, that is a difference of three seconds a message, and three seconds times five thousand is more than four hours of waiting a day when the messages run one after another. If the large model costs several times the small one per message, the cost gap is also multiplied by five thousand, every day. Cost and speed, per call, become the whole budget of a repeated task, so for a task that repeats you pick the lightest model that does each part of it well [1].

So the order of work is the one you followed in this lesson. Say what the task needs. Run a few real inputs on each candidate and fill the table. Then read the table at the volume you expect, and measure before you schedule anything repeated. The large model goes where the table says a cell changed from no to yes, and nowhere else.

Checkpoint · scenario

Five thousand messages a day

A colleague wants to label the team’s five thousand daily customer messages with the large model, “because it is the best one”. What do you propose?

Agree. The large model gives the best labels, and a few extra seconds per message is nothing.Run fifty real messages through both models first, compare the labels, and time each run before choosing.Use the small model without a test. It handled five messages in the lesson, so it will handle five thousand.Use the large model for the first week and switch to the small one once the labels look right.

DecideHintSkip

Pitfall: Leaving the picker on the largest model

A team sets every tool to the largest model, because nobody wants to be the one who chose a worse answer. Six months in, the largest model is labeling tickets, reformatting exports and answering questions about the office printer. A test on twenty tickets shows the small model gives the same labels in a quarter of the time. Nobody had run the comparison, because “the best” never felt like a decision. Treat the model pick as a decision, and make it from a table you filled in.

Exercise

Open your chat tool and find the model picker. Run the three inputs from the first section on the small model, each in a fresh chat, and then on the large model, again in fresh chats. For each of the six answers write down how long it took and whether you would use it as it is. This takes about ten minutes and leaves you with a six-cell table. A good result is a table where you can point at each “no” and say what was missing from the answer, and at each pair of “yes” cells and say how much longer the large model took. Where did your predictions from the first section turn out wrong, and what does that tell you about your default?

Stretch: Add a fourth input that asks about something that happened this month, run it on both models, and note whether either one says it cannot know, or whether the answer needs a model with a search tool instead.

Recap

1. Say what a task needs before you name a model: reformatting, classification, multi-step reasoning, or fresh facts.
2. Find the model fit by running a few representative inputs on each candidate. The size of a model doesn’t tell you its fit on your task.
3. A small model is often good enough for reformatting and classification, and a large one pays off on an ambiguous request. Fresh facts need a model with a search tool, whatever its size.
4. Cost and speed are per call. At the volume of a repeated task they become the whole budget, so measure before you schedule.
5. Leaving the picker on the largest model is a choice too, and the table is how you check it.

You can now

• Picks a model for a task by testing fit, cost and speed

Mark lesson finishedI know this, skip it

References

1. Addy Osmani, Ivar Soares Urdalen, Leo Simons. How agents think: tokens and context, reasoning strategies, model choice, system prompts. Agent Engineer Course. Course. AEC-02