Black vs Is Your AI Hallucinating?

A black-box evaluation of how AI-generated tests find functional bugs in live APIs.

How our white-box proxy model gives you a per-token hallucination score, including exactly what it costs.

About Black

A black-box evaluation of how AI-generated tests find functional bugs in live APIs. The harder question is whether those tests find bugs. Each system receives only a JSON schema and one valid sample payload, then must generate API test cases that expose failures in a live reference API. The evaluation uses APIEval-20 v1.0, a black-box benchmark contributed by KushoAI. Because KushoAI is also one of the evaluated systems, this report includes the methodology, workflow definitions, repeated-run setup, and robustness checks so readers can understand where the performance difference comes from.

About Is Your AI Hallucinating?

How our white-box proxy model gives you a per-token hallucination score, including exactly what it costs. We ship an API that returns a hallucination score for every token that a frontier language model generates. Before describing how it works, here is the most important thing about it: the percentage score that we return is not actually the probability that the token is wrong. The obvious problem is that frontier models do not expose the hidden states that we would need to probe to reveal their thoughts.