Black vs Supercov

A black-box evaluation of how AI-generated tests find functional bugs in live APIs.

Supercov turns uncovered paths into the next focused test for coding agents, then verifies the improvement with your existing test suite.

About Black

A black-box evaluation of how AI-generated tests find functional bugs in live APIs. The harder question is whether those tests find bugs. Each system receives only a JSON schema and one valid sample payload, then must generate API test cases that expose failures in a live reference API. The evaluation uses APIEval-20 v1.0, a black-box benchmark contributed by KushoAI. Because KushoAI is also one of the evaluated systems, this report includes the methodology, workflow definitions, repeated-run setup, and robustness checks so readers can understand where the performance difference comes from.

About Supercov

Supercov turns uncovered paths into the next focused test for coding agents, then verifies the improvement with your existing test suite. Before a reset, or overnight, turn idle agent time into coverage that stays with the repository. Every pass returns fresh, executable evidence, so agents can keep iterating around the clock while regressions are caught before they ship. Write first test Use coverage to choose one useful test. Start a new task in the repository, paste the prompt, and send.