Loading...
Loading...
Every vendor deck this year claims frontier LLMs can run the SOC. Almost none of them show the test. We built one, ran the top models through it, and the results reset the conversation: the best model finishes about two-thirds of a real investigation, and the newest model isn't always the best.
For the first time, we've built a benchmarking framework designed specifically for AI SOC work. Existing cybersecurity benchmarks like CyberSecEval and CTIBench test knowledge or narrow skills. Ours tests the whole job. The difference is realism and depth.
Forget generic scenarios. Our benchmark runs autonomous investigation across 100 full-kill-chain cases that mirror what human SOC analysts see every day. Each case is grounded in known malicious activity with a clear ground truth, so an AI agent's work can be graded against a fixed baseline rather than a vague rubric.
The scenarios draw on historical behaviour from well-known APT groups (APT32, APT38, APT43) and cybercriminal crews (Cobalt Group, Lapsus$). They cover a wide slice of MITRE ATT&CK tactics and techniques, with heavier weighting on the threats most SOCs actually see: ransomware and phishing.
To run the tests, we drove every model through our Simbian AI SOC Agent. Evaluation is evidence-grounded and data-driven. Each run starts with a triggered alert, exactly the way an analyst's shift starts. From there, the agent has to decide whether the alert is a true positive or a false positive, surface evidence of malicious activity (think "CTF flags" in a red-team exercise), interpret that evidence by answering security-contextual questions, and produce a high-level narrative of what the attacker did. This evidence-based approach is critical for keeping hallucinations in check. The LLM isn't guessing; it's validating its own reasoning against the trail.
We benchmarked some of the most well-known and high-performing models available as of May 2025, including models from Anthropic, OpenAI, Google, and DeepSeek. The results are promising, with caveats. Every high-end model completed more than half of the investigation tasks, landing between 61% and 67%. For reference, the best human analysts at the first AI SOC Championship, working with an AI SOC on their side, scored between 73% and 85%. Simbian's own AI SOC Agent, dialled to extra effort, reached 72%. That gap matters. It says LLMs are capable of far more than summarising and retrieving data. They can triage alerts and drive tool use through API interactions. It also says the best humans, augmented, still finish ahead of the best autonomous models. Both facts are true at the same time.
The first big lesson from the runs is a boring one on the surface, and a serious one underneath: prompt engineering and agentic-flow engineering carry real weight. Feedback loops, retries, and continuous monitoring aren't optional polish. In early runs, several models stumbled until we tightened the prompts and added fallback paths for the coding agents that parse retrieved data. Which leads to a second lesson. AI SOC applications lean heavily on the software-engineering capabilities of the underlying LLM. Investigation work is coding work in disguise: parse this JSON, walk this graph, correlate these fields, call this API. Models that write good code investigate better.
The third finding surprised us. Sonnet 3.5 sometimes outperformed the newer Sonnet 3.7 and 4.0. We suspect "catastrophic forgetting". Further domain specialisation for software engineering or instruction following may quietly degrade the cybersecurity knowledge and investigation planning we care about most. The practical takeaway: newest is not synonymous with best for defensive work. Benchmarking has to catch this, because release notes won't.
Finally, we found that "thinking models" — the ones that use post-training techniques and internal self-talk — didn't show a meaningful edge in AI SOC work. Every tested model landed in a similar band. That echoes findings from studies on software bug fixing and red-team CTF applications, which suggest that once an LLM hits a capability ceiling, extra inference buys marginal improvement at real cost. The implication is direct. Human-validated LLM applications matter in the SOC, and so does the continued build-out of fine-grained, specialised benchmarks that push cybersecurity reasoning forward rather than benchmarking general reasoning and hoping the security skill comes along for the ride.
A 67% autonomous score is impressive on paper and dangerous in production. It means one in three investigations is incomplete, mishandled, or wrong. In a SOC that sees tens of thousands of alerts a week, that residual error rate isn't a rounding error. It's the difference between catching the intrusion and reading about it later.
That's the whole reason our AI SOC Agent doesn't ship as a bare LLM behind an API. The Agent wraps the model in a harness that closes the gap: grounded evidence retrieval through Context Lake™, structured tool use, retry logic, disagreement checks between agents, and a human-in-control layer that keeps containment authority and escalation calls with the analyst. Self-improving, not self-driving. The benchmark tells us where the model plateaus. The product is what turns that plateau into resolved cases.
If you're comparing AI SOC vendors, three practical asks fall out of this work.
The AI SOC LLM Leaderboard measures LLMs using Simbian's AI SOC Agent, a proven framework for putting AI to work inside the SOC. The AI SOC Agent is deployed at some of the largest SOCs in the world. In the recent AI SOC Championship 2025, which drew more than 100 analysts globally, the AI SOC Agent outperformed 95% of participants at correctly investigating alerts with supporting evidence.
For the full details and results of the benchmarking, see the leaderboard. We'll use this benchmark to evaluate new foundation models on a regular cadence and share what we find. A deeper write-up of the methodology and findings is queued for an upcoming cybersecurity conference.
