What the agent did
An agent turns client call transcripts into Asana tickets. He ran a blind test of two models on 10 real calls and 30 real tickets, then rewrote the instructions to cut made-up details.
Result, as claimed
On 10 real client calls graded blind, Claude Haiku 5.5 caught 83 of 92 promises against 52 for Gemini 3.8 Flash, at $0.0095 a call against $0.037, cutting that bill 74%. On 30 tickets it won 24, at $0.0010 a ticket against $0.0075. Rewriting the instructions cut made-up details from 81 to 14.
This is the result as claimed in the public source linked below. It has not been checked and is not a typical outcome.
Proof
Try it yourself
Blind-test two AI models on the same real task before you choose one.
Prompt
I want to compare two AI models on this task: [describe it, for example pull every promise from a call transcript]. Here are [10] real examples and, for each, the correct answer I wrote myself: [paste]. Write a short scoring sheet I can use to grade outputs without knowing which model wrote them: what counts as correct, missed and made up. Then tell me how to label the outputs A and B so I grade blind.
Tools
Steps
- Pick 10 real examples and write the right answers yourself.
- Run both models with the same instructions.
- Grade the outputs blind with the sheet.
- Compare cost per task and switch only if the cheaper one is as good.
Watch out: Ten examples is a small test. Re-run it when your instructions or the models change.
Starter version by Solopreneur Code
Skill: Build with a coding agent
Add this skill
Blind model test
Sets up a small blind test of two AI models on your real task so you pick on quality and cost, not habit.
- Works with
- Claude, ChatGPT, Gemini Works with any AI that can read files you upload.
- Tools in the story
- Claude Haiku 5.5, Gemini 3.8 Flash, Asana
- Country
- United States
How to use this: Copy the skill, paste it into Claude, ChatGPT or Gemini, and ask it to save it as a skill. If it cannot, it will still follow the steps.
Show skill
If the Copy button does not work, select this text and copy it.
Job: turn meetings into notes and documents
Other tools for this job
Tools used
- Claude Haiku 5.5See official pricing: Claude Haiku 5.5, opens in a new tab
- Gemini 3.8 FlashSee official pricing
- AsanaSee official pricing: Asana, opens in a new tab
Proven in other stories
- A Singapore agency turned client meetings into handover documents with MeetGeek and Claude, and says onboarding got 85% fasterRecords client meetings and turns them into handover documents.
Could also do this
- Fireflies.ai Not tested in a storySays it transcribes meetings and gives notes, action items and summaries after each meeting.
- Fathom Not tested in a storyAI notetaker. Says it summarises meetings and gives searchable transcripts and follow-ups.