---
title: "An agency CTO blind-tested a cheaper AI model on 10 client calls and cut one AI bill by 74% while catching more promises"
person: "Micah Berkley"
handle: "@MicahBerkley"
profile: "https://x.com/MicahBerkley"
tools:
  - "Claude Haiku 5.5"
  - "Gemini 3.8 Flash"
  - "Asana"
country: "United States"
category: "Save money"
category_slug: save-money
proof:
  - label: "Read the post on X"
    url: "https://x.com/MicahBerkley/status/2115799364576612830"
date: 2026-11-02
date_posted: 2026-10-29
as_claimed: "The result is shown as claimed in the public source linked here. It has not been checked and is not a typical outcome."
url: "https://agentstories.org/stories/micah-berkley-blind-model-test-cut-bill.html"
markdown_url: "https://agentstories.org/stories/micah-berkley-blind-model-test-cut-bill.md"
site: "Agent Stories by Solopreneur Code"
---

# An agency CTO blind-tested a cheaper AI model on 10 client calls and cut one AI bill by 74% while catching more promises

Micah Berkley @MicahBerkley · United States · Claude Haiku 5.5, Gemini 3.8 Flash, Asana

## What the agent did

An agent turns client call transcripts into Asana tickets. He ran a blind test of two models on 10 real calls and 30 real tickets, then rewrote the instructions to cut made-up details.

## Result, as claimed

On 10 real client calls graded blind, Claude Haiku 5.5 caught 83 of 92 promises against 52 for Gemini 3.8 Flash, at $0.0095 a call against $0.037, cutting that bill 74%. On 30 tickets it won 24, at $0.0010 a ticket against $0.0075. Rewriting the instructions cut made-up details from 81 to 14.

> The result is shown as claimed in the public source linked here. It has not been checked and is not a typical outcome.

## Proof

- [Read the post on X](https://x.com/MicahBerkley/status/2115799364576612830)

## Try it yourself

Blind-test two AI models on the same real task before you choose one.

Prompt:

```text
I want to compare two AI models on this task: [describe it, for example pull every promise from a call transcript]. Here are [10] real examples and, for each, the correct answer I wrote myself: [paste]. Write a short scoring sheet I can use to grade outputs without knowing which model wrote them: what counts as correct, missed and made up. Then tell me how to label the outputs A and B so I grade blind.
```

Tools:

- [Claude](https://claude.ai): Haiku 5.5 won his test
- [Gemini](https://gemini.google.com): The model it was compared with

Steps:

1. Pick 10 real examples and write the right answers yourself.
2. Run both models with the same instructions.
3. Grade the outputs blind with the sheet.
4. Compare cost per task and switch only if the cheaper one is as good.

Watch out: Ten examples is a small test. Re-run it when your instructions or the models change.

## More

- [This story as a web page](https://agentstories.org/stories/micah-berkley-blind-model-test-cut-bill.html)
- [More stories: Save money](https://agentstories.org/categories/save-money.html)
- [More stories from United States](https://agentstories.org/countries/united-states.html)
- [More stories that use Asana](https://agentstories.org/tools/asana.html)
- [Every story, indexed for AI assistants](https://agentstories.org/llms.txt)
