|
I asked a well-known API for my bounced emails. It
answered 200 with five emails, and none of them had
bounced.
SilentProbe
describes what an agent gets back in that case: "HTTP
200 with a parsable body, no exception to catch and no
field to branch on."
The study tested 27 vendors' live APIs with 219
deliberately wrong requests, then gave twelve models
ordinary tasks against them:
-
Across 2,501 public OpenAPI documents, only 15.2% of
parameters declare a constraint a machine can check.
-
When the constraint was in the schema, a schema
validator in front of the APIs returned an honest
error 111 times out of 111. It caught 110 of those
before the request reached the vendor.
-
When it was only in the description, the API failed
silently 44 times out of 61.
-
Agents noticed the silent failure 12% of the time, and
fixed it 0% of the time.
-
In 41% of cases they told the user something
didn't exist when it did (28% with no hedge). In
12% they made up a figure.
In the authors' words, "constraint form, not vendor identity, predicts
honesty". The agents did what agents do with a 200: they
believed it.
I've been running agents and fixed checks against two
APIs, which I'll call A and B. These are the five
ways their HTTP APIs misled my agents, ranked by damage,
worst first.
1. It says 200 to a filter it ignored
That was API A, answering
GET /emails?limit=5&status=bounced
in 161 ms: 3 queued, 2 delivered, and nothing in the
response saying the filter was ignored. An agent would
hand that list to your user as their bounces.
I'm not the only one. The Colony is a forum and
social network for AI agents. In August, one of its
agents, ANP2 Network,
probed the site's own API. Notification objects there carry a field named
is_read, so it tried the obvious filter first:
?is_read=false
returned 200 and all 50 rows. The parameter that works is
?unread=true, which returned 2. The same API answered
?sort=NOT_A_SORT
with an honest 422. Its conclusion: "Strictness is a
property of an individual parameter." In October,
Nevermore
followed up
with
GET /claims?status=banana: 200, the same row as without the filter.
Nevermore's verdict: "the filter was never
read".
2. It accepts a field it doesn't know, then drops it
I sent API A a
POST /emails
with
"unknown_field": true. 200, an id, the email went out. Fine for a made-up
field. Now picture an agent typing
reply_too
instead of
reply_to: same 200, and your replies go nowhere.
Zod does this by default:
z.object()
strips unknown keys, which is how Shubham
Shrivastav's API
returned 201
"for a request whose main instruction it had thrown
away".
3. Its responses break its own spec
API B lets you ask for only the fields you need, with a
fields=…
query parameter. Its spec also promises that some
properties are always there: on a record,
name,
title
and
emails
are marked
required. Ask for a smaller set of fields and those
"required" properties are missing from the
response.
I ran an automated test of API B against its own spec. It
found 9 responses that broke the spec, and every one of
the 9 came from a request with
fields. An agent that trusts the spec now holds objects that,
according to that spec, can't exist.
4. It calls a bad key a bad request
HTTP has separate codes for two separate problems. A 400
Bad Request means "fix your request". A 401
Unauthorized means "fix your credentials". An
agent needs to know which one it's facing, because
the fixes have nothing in common.
API A gets it right when there's no key at all:
{"statusCode":401,"message":"Missing API Key","name":"missing_api_key"}
Send a wrong key instead, and listing emails or domains
returns this:
{"statusCode":400,"message":"API key is invalid","name":"validation_error"}
Try to send an email with the same wrong key and you get a
401, still named
validation_error. Same mistake, two status codes depending on the
endpoint, and both times the name says the request failed
validation.
In my run the model read "API key is invalid",
reported it and stopped. The code around it is what
breaks: the script the agent writes and the retry logic.
That code branches on the status and the name, because
that's what they're for. Handle auth errors with
a check for 401 and you catch the dead key when sending,
but miss it on every list call, where the same key comes
back as a 400 and gets treated as a bad request.
5. It answers with someone else's error
I asked my agent to schedule a reminder email on API A,
then cancel it. Its first cancel attempt said "Email
is not scheduled" while the email sat in the queue,
so it dropped to raw HTTP and wrote a short Python script
with urllib. The cancel and a GET to check the email both
came back with this, as plain text:
403 error code: 1010
The text comes from Cloudflare, which sits in front of API
A and turns away requests that have no User-Agent header.
urllib doesn't send one by default. The body
isn't JSON, has no error name, and doesn't
mention the header. API A does document the User-Agent
requirement. The error just doesn't point to it.
My agent got past it on the next try. It added
'User-Agent': 'curl/8.7.1'
to its headers, passing itself off as curl, and both calls
came back 200. A 403 usually means "you're not
allowed", and an agent that took this one at face
value would have told you it had no permission to cancel
its own email.
A 400 is a good day
Multigrid
sorts parameters
into four kinds: rejected, honoured, dropped and
approximated. A dropped parameter is
"indistinguishable from honoured without a test
designed to tell them apart". Their line: "A 400
is a good day."
For agents, fail loudly. A 400 gives the agent something
to fix. A 200 lets it tell your user something that
isn't true.
Next week: how to fix all five, and which of those fixes
can break your existing clients.
Subscribe to my newsletter so it lands in your
inbox.
Thanks for reading :)
Unsubscribe
| 1 Bd des clubs - CFC, 20200 Casablanca, Morocco
|
