Black vs Winnow

A black-box evaluation of how AI-generated tests find functional bugs in live APIs.

Winnow scores every post in your X timeline as you scroll and highlights the few worth replying to. Local, deterministic, read-only.

About Black

A black-box evaluation of how AI-generated tests find functional bugs in live APIs. The harder question is whether those tests find bugs. Each system receives only a JSON schema and one valid sample payload, then must generate API test cases that expose failures in a live reference API. The evaluation uses APIEval-20 v1.0, a black-box benchmark contributed by KushoAI. Because KushoAI is also one of the evaluated systems, this report includes the methodology, workflow definitions, repeated-run setup, and robustness checks so readers can understand where the performance difference comes from.

About Winnow

Winnow scores every post in your X timeline as you scroll and highlights the few worth replying to. Local, deterministic, read-only. Why this exists I was replying to the wrong posts. I used to reply to whatever showed up on the timeline, usually posts that were already ten replies deep, or three hours cold by the time I got to them. Same replies, just aimed at posts that were actually still open. The same reply is worth a great deal in the first fifteen minutes of a post with room left in it, and almost nothing on a post that already has forty replies and has finished being read.