Black vs Project Huginn

A black-box evaluation of how AI-generated tests find functional bugs in live APIs.

Train and fine-tune AI models at 50% lower cost. Distributed GPU sharing platform with free base models (LLaMA 3.1, Mistral 7B, Phi-3, CodeLlama, Gemma, Qwen). LoRA, QLoRA...

About Black

A black-box evaluation of how AI-generated tests find functional bugs in live APIs. The harder question is whether those tests find bugs. Each system receives only a JSON schema and one valid sample payload, then must generate API test cases that expose failures in a live reference API. The evaluation uses APIEval-20 v1.0, a black-box benchmark contributed by KushoAI. Because KushoAI is also one of the evaluated systems, this report includes the methodology, workflow definitions, repeated-run setup, and robustness checks so readers can understand where the performance difference comes from.

About Project Huginn

Train and fine-tune AI models at 50% lower cost. Distributed GPU sharing platform with free base models (LLaMA 3.1, Mistral 7B, Phi-3, CodeLlama, Gemma, Qwen). LoRA, QLoRA & full fine-tuning. Built-in Model Studio and AI playground. 1 HU = €0.21. Alternative to cloud GPU rental. Earn from any device. Startup Denmark approved. You choose the protection level on every job: Shield+ keeps your data encrypted and split so no machine ever sees the whole, or Vault runs on sealed hardware no one else can read.