HackerRank's Open-Source ATS Scored One Resume From 66 to 99

HackerRank has open-sourced its applicant tracking system, and a developer who tested it found the same resume scoring anywhere from 66 to 99 out of 100 depending on the run. The project, published on GitHub as interviewstreet/hiring-agent, spread through LinkedIn and Reddit posts before the author of the danunparsed blog tried it.

What the test found

The first working run gave the author's resume 90 out of 100. After cleaning up debug print statements left over from setup, the same resume with the same command scored 74. Disabling development mode and running the tool in a loop a hundred times produced scores from 66 to 99; with a company cutoff at 85, the same resume fails about 65 percent of the time.

The variance is not uniform. Technical skills scored 8 out of 10 in 98 of the 100 runs, because that category is essentially a checklist. Projects, worth 30 points, fluctuated constantly — the model sometimes said projects lacked architectural complexity and sometimes praised real-world deployment. A GitHub issue from October reported scores of 27, 34, 32, 34, 34 and 30 across six consecutive runs. Experience, worth 25 points, returned 25 out of 25 every single time, including for an old resume with a single internship, because its prompt is two lines long with no rubric or examples. As the author puts it: consistent but useless.

How the scoring works

The tool parses a PDF resume into text, then calls an LLM six times to extract structured data — basics, work history, education, skills, projects and awards — pulls the candidate's GitHub profile and top repositories as extra context, and feeds everything into a final grading call. The score is out of 100 with up to 20 bonus points: 35 points for open source contributions, 30 for personal projects, 25 for work experience, 10 for technical skills, plus bonuses for startup experience, a portfolio site or a technical blog. The default model is gemma3:4b at temperature 0.1.

Why it matters

Switching to a stronger model tightened the range without fixing the problem: Gemini clustered scores between 48 and 64, which still fails 28 percent of the time against a cutoff of 60, and running Claude Opus 4.8 narrowed the spread only slightly. The author argues that the non-determinism is a design flaw rather than a tuning issue, and that grading judgment calls is a poor fit for language models. A correction on June 28 noted that the evaluation template opens with the words Software Intern, though re-running with an explicit senior-engineer prompt gave identical results, suggesting the scoring dimensions do not depend on the position. The post also warns that open source and personal projects carry 65 percent of the weight, which can sideline experienced engineers whose work never reached GitHub.