Module 2 ยท Lab 2 ยท 20 min

๐Ÿ’ฌ Analyze and Optimize an AI Application

A production customer-support assistant, three Claude agents deep, and a bill nobody has examined.

Your Scenario

You have inherited a customer support assistant that is already in production. Every ticket passes through three Claude agents in sequence: 5,000 tickets a month, $5,000 on the bill, exactly $1.00 a ticket. Nobody has looked at that number since the system was built, and your CFO has just worked out it is 2% of revenue.

Your job is to bring it down without breaking it. One constraint holds all the way through: resolution must never fall below 85%. The sections below walk you from understanding the bill to designing a replacement.

  1. Your Scenario โ€” you are here. How the application is wired, and which agent spends what.
  2. Understand the Baseline โ€” why one of the three agents carries 42% of the bill, and what swapping its model would cost you in resolution.
  3. Optimization Levers scored โ€” cut at least $1,500 a month without dropping below 85% resolution.
  4. Scaling Challenge scored โ€” decide what has to change when volume grows ten times.
  5. Design Your System scored โ€” build one from scratch, to a budget that tightens per ticket as volume grows.

The first two sections are reading. The last three are scored, and the sidebar keeps the count as you go.

Open full size โ†’
Monthly Cost
$5,000
Per Ticket
$1.00
Response Time
90 sec
Resolution
87%
Agent 1 (Triage)$1,400 (28%)
Agent 2 (Analysis)$2,100 (42%)
Agent 3 (Resolution)$1,500 (30%)
Total Monthly$5,000
๐Ÿ’ก Where the money is. Agent 2 takes 42% of the bill, half as much again as Agent 1 โ€” and not because it runs more often, since all three see every ticket. The next section explains what actually makes it expensive.

Understand the Baseline

Goal: analyze where money goes. No changes yet.

Question 1 ยท Which agent costs most?

Answer: Agent 2 (Analysis) at $2,100/month = 42% of budget. Why? It has the longest context (ticket + category + history + similar cases) โ€” more context means more input tokens, and Opus input is $15/1M tokens.

Question 2 ยท What's your cost per ticket?

Answer: $1.00/ticket (5,000 tickets / $5,000). Tokens per ticket: 2,100 total (1,700 in + 400 out).

Question 3 ยท What if you switch Agent 2 to Sonnet?

Current (Opus)
Agent 2 Cost$2,100
Quality95%
Resolution87%
If Sonnet ($3/1M)
Agent 2 Cost$420
Quality90%
Resolution85%
Savings
-$1,680/mo
๐Ÿ’ก Key insight: you could save $1,680/month by switching to Sonnet, but you'd lose 2% resolution rate. That's the tradeoff.

Exercise 1 ยท Optimization Levers

Challenge: reduce costs 30% without dropping below 85% resolution.

Real-time results

New Cost
$5,000
Savings
-$0
Resolution
87%
Status
โœ… Target Met!
Before โ†’ After
Cost$5,000 โ†’ $5,000
Per ticket$1.00 โ†’ $1.00
Resolution87% โ†’ 87%
๐Ÿ’ก Hint: try Haiku for Triage (simple categorization) + Sonnet for Analysis & Resolution to hit 30% savings.

Exercise 2 ยท Scaling Challenge

Challenge: your optimized system scales to 50K tickets/month (10x growth). What changes?
At 5K/mo
$3,500
At 50K/mo
$35,000
New Problems
Quotas, SLA, Infra

Choose your scale strategy

๐Ÿ‘† Click a strategy above to see the impactโ€ฆ

๐Ÿ’ก Reality check: scaling 10x isn't just multiplying by 10 โ€” you hit API quotas, need infrastructure changes, and operational complexity increases. Your architecture needs to adapt.

Exercise 3 ยท Design Your Optimal System

Challenge: design from scratch. Pick your volume, then fit the budget that comes with it โ€” $6K at 5K tickets, $11K at 10K, $45K at 50K. Min resolution: 85%. Note the allowance per ticket falls as you scale, from $1.20 to $0.90: the business expects volume to buy efficiency.

Your design costs

Est. Cost
$10,500
Est. Quality
90%
Budget
โŒ Over
Design vs. constraints
Budget limit$6,000
Your cost$10,500
Min resolution85%
Your est.90%
Status
โŒ Over Budget