Save moneyBusiness size: Small team

An agency CTO blind-tested a cheaper AI model on 10 client calls and cut one AI bill by 74% while catching more promises

Micah Berkley @MicahBerkley · United States · Claude Haiku 5.5, Gemini 3.8 Flash, Asana

Go to the skill

What the agent did

An agent turns client call transcripts into Asana tickets. He ran a blind test of two models on 10 real calls and 30 real tickets, then rewrote the instructions to cut made-up details.

Result, as claimed

On 10 real client calls graded blind, Claude Haiku 5.5 caught 83 of 92 promises against 52 for Gemini 3.8 Flash, at $0.0095 a call against $0.037, cutting that bill 74%. On 30 tickets it won 24, at $0.0010 a ticket against $0.0075. Rewriting the instructions cut made-up details from 81 to 14.

This is the result as claimed in the public source linked below. It has not been checked and is not a typical outcome.

Proof

Read the post on X

Try it yourself

Blind-test two AI models on the same real task before you choose one.

Difficulty: Advanced

Prompt

I want to compare two AI models on this task: [describe it, for example pull every promise from a call transcript]. Here are [10] real examples and, for each, the correct answer I wrote myself: [paste]. Write a short scoring sheet I can use to grade outputs without knowing which model wrote them: what counts as correct, missed and made up. Then tell me how to label the outputs A and B so I grade blind.

Tools

  • Claude Haiku 5.5 won his test
  • Gemini The model it was compared with

Steps

  1. Pick 10 real examples and write the right answers yourself.
  2. Run both models with the same instructions.
  3. Grade the outputs blind with the sheet.
  4. Compare cost per task and switch only if the cheaper one is as good.

Watch out: Ten examples is a small test. Re-run it when your instructions or the models change.

Starter version by Solopreneur Code

Skill: Build with a coding agent

Add this skill

Build with a coding agentDifficulty: Advanced

Blind model test

Sets up a small blind test of two AI models on your real task so you pick on quality and cost, not habit.

Works with
Claude, ChatGPT, Gemini Works with any AI that can read files you upload.
Tools in the story
Claude Haiku 5.5, Gemini 3.8 Flash, Asana
Country
United States

How to use this: Copy the skill, paste it into Claude, ChatGPT or Gemini, and ask it to save it as a skill. If it cannot, it will still follow the steps.

Show skill

If the Copy button does not work, select this text and copy it.

Job: turn meetings into notes and documents

Other tools for this job

Tools used

Proven in other stories

Could also do this

Sign up for the newsletter

Get the next story from Solopreneur Code in your inbox.

Form not loading? Subscribe on Substack

Back to all stories

More like this

Browse all stories
Make moneyBusiness size: CompanyIncludes prompt

A Washington roofing company rebuilt its Zapier workflows with Claude and says its automations lifted inquiry-to-job conversion by at least 15%

Ben Beckett · United States · Zapier, Zapier Next-Gen Zaps, Claude, Jobber, Asana, CallRail

Fixed Zaps copy leads, update job cards, file estimates, text new leads back within minutes and flag customers nobody called back. Claude built 14 Next-Gen Zaps, which Zapier monitors and suggests fixes for.

Result, as claimed

About 12,500 automated steps a month, which he estimates at 50 to 100 hours a week. Inquiry-to-signed-job conversion up at least 15% across all automations. Follow-up Zaps catch 10 to 15 leads a month. The ops app saves about 15 hours a week, roughly $30,000 a year at bookkeeper rates.

Save moneyBusiness size: Small teamIncludes prompt

A Nassau walking tour owner used Claude to build 40 Zapier workflows and expects outsourced support costs to fall from about $20,000 to about $1,200 a year

Jaime Lewis · Bahamas · Claude, Zapier Next-Gen Zaps, Slack, Google Forms, Google Sheets, QuickBooks Online, Jotform, Notion

Claude converted his classic Zaps and wrote new Next-Gen Zaps from his instructions, and wrote documentation for each into a Notion registry. Running Zaps read guides' receipts, nudge guides who have not sent one, and validate booking rows hourly before they post to QuickBooks.

Result, as claimed

40 Next-Gen Zaps built. Nearly 700 runs in the past four weeks at a 98.7% success rate. Outsourced support cost of about $20,000 a year is expected to fall to about $1,200, roughly $19,000 saved (projection).

Save moneyBusiness size: Small teamIncludes prompt

A Singapore accounting outsourcing firm built its own AI system and says it frees up 396 professional hours a month

Accountingbpo Pte Ltd · Singapore · In-house AI accounting system, models not named

An in-house AI system covers document processing, work coordination, reconciliation, financial reporting and analysis.

Result, as claimed

IMDA's award citation says document processing time fell by up to 67% and 396 professional hours are freed each month.