Home / Lessons / Judge the result / 21 of 27

Over ten times the estimate

My own count said one thing. The provider's count said another.

  1. My own estimate of what the tests used: 9,000 units at most.
  2. The automated checks showed nothing. They had no key, so they made no live calls.
  3. The provider counted 97,000.
  4. More than ten times my estimate. I believe the party that sends the bill.

The provider's export showed live calls that my own estimate had missed. Local test runs had a service key and were making real calls. The automated checks never showed the leak, because their environment had no key at all.

It changed which evidence I trust. The provider's own record now outranks any reconstruction of usage that I build myself. Tests block live provider calls at the client boundary, and every allowed request records the units the provider reports.

How I do it

The provider's record is the source of truth for usage. My internal numbers are a guess until they match it.

The rule

Measure at the boundary where the money is spent, and believe the party that sends the bill.

The skill I use for this: /systematic-debugging. Made by Jesse Vincent.

Read next

The ideas behind it

This lesson as plain text, for your agent. Or all 27 on one page.

Try “secrets”, “merge”, “handover” or “database”.