Black vs Memsprout

A black-box evaluation of how AI-generated tests find functional bugs in live APIs.

Checkout copy is owned by Growth, ship behind the `pricing_v2` flag.

About Black

A black-box evaluation of how AI-generated tests find functional bugs in live APIs. The harder question is whether those tests find bugs. Each system receives only a JSON schema and one valid sample payload, then must generate API test cases that expose failures in a live reference API. The evaluation uses APIEval-20 v1.0, a black-box benchmark contributed by KushoAI. Because KushoAI is also one of the evaluated systems, this report includes the methodology, workflow definitions, repeated-run setup, and robustness checks so readers can understand where the performance difference comes from.

About Memsprout

Checkout copy is owned by Growth, ship behind the `pricing_v2` flag. Works with the tools your team already uses Claude Code. The context gap Every agent on your team is working blind. Each person stores what they know as they work, decisions, gotchas, domain context. Everything you capture, a decision, a gotcha, a how-to, is one Memory. A Space scopes who can see a Memory: your own Personal Space, or a Shared Space with owner, editor, and viewer roles.