A researcher built a vulnerable app and paid LLMs $1,500 to try to hack it

A security researcher who tests apps for authorization flaws built a deliberately vulnerable book-review app, paid large language models to break it, and then published the results and the transcripts.

A hardened API with a soft data layer

The target, called BookNook, is a mock React Native app built in Expo with a Python backend running FastAPI. Each model was asked to find a flag hidden in a user's private reviews. The API itself is well protected, but it stores data in Firebase, and a google-services.json file shipped inside the APK exposes the project's Firebase configuration. The intended path was to sign up directly against Firebase and then read the Firestore database. The author notes this is the same category of flaw — broken access control, or missing object-level authorization — that he has found in real Firebase and Supabase apps.

Solve rates and costs

Each run had a $10 budget and a two-hour limit. GPT-5.5 solved 7 of 10 attempts, at an average of $6.62 per run and $9.46 per solve. DeepSeek V4 Pro solved 3 of 10 for $0.19 per run and $0.62 per solve. Claude Sonnet 4.6 and Claude Opus 4.8 each solved 2 of 10, at $9.15 and $3.23 per run respectively. Five models scored zero: DeepSeek V4 Flash, Gemini 3.1 Pro Preview, Gemini 3.5 Flash, MiniMax M2.7 and Step 3.7 Flash. Models tested fewer than ten times included GLM 5.1 (1 of 4) and Kimi K2.6 (1 of 1).

How the models behaved

Almost every successful GPT-5.5 run unzipped the APK and moved straight to Firebase. Half of the DeepSeek V4 Pro runs never examined Firebase at all, focusing only on the API or the app. Several Claude Sonnet 4.6 runs were on the right track when the budget ran out, while Opus sessions were ended early by safety guardrails — late refusals rather than immediate ones. Gemini 3.1 Pro Preview refused at once, visible in a median of just 9,000 tokens per run against 100,000 or more for the models that tried. Step 3.7 Flash mapped the API carefully but wrongly claimed to have found exploits.

Limits of the experiment

The author says the test is not a scientific evaluation: he aimed for ten runs per model, stopped after spending $1,500, and says he should have stopped sooner. OpenAI had already approved his account for security research, which is why GPT produced no refusals. He also reports repeated API outages at MiniMax and GLM, and says running the harness on Modal preempted about 10% of runs, which he would replace with AWS next time. The challenge archive is public for anyone who wants to test their own models.