Quick disclosure before anything else: I work at DeepTempo, and one of the models in this benchmark (LogLM) is ours. So yeah, factor that in as you read. The upside is that all of it is open source and reproducible, which means you don't have to trust me on a single number here. Clone it, run it, tell me where I'm wrong. That's the whole reason it's public.
I've been quietly annoyed for a while now. Every "AI in the SOC" pitch I see opens with a gorgeous demo and somehow never gets around to showing how the thing holds up on the boring, noisy telemetry a defender stares at all day. So I finally built a benchmark for exactly that (SOCBench), and I started with the least glamorous but most challenging SOC task there is: detection on raw NetFlow.
Here's the part I want to be upfront about: I rigged the setup in the LLMs' favor, on purpose.
\* The three frontier models got to run as full multi-turn agents. Bounded ReAct loop, read-only investigative tools, four expert personas, big context budgets, and a cost cap so they couldn't run forever.
\* LogLM got none of that. It's a small encoder-only model, and all it ever saw was the raw flows. One shot, no tools, no personas. Here's the traffic, what's malicious?
\* Everyone got the same 1,205 eval units (Stratosphere Labs captures), the same hidden ground truth, and it was all zero-shot.
The logic was simple. If the LLMs were going to fall over, I wanted them to do so under the most flattering conditions I could create — every advantage stacked on their side, and our little encoder walking in with nothing but the flows.
So what happened?
\* They can tell when something's off. Verdict F1 (just "is this unit malicious or not") came in between 0.86 and 0.93 for each model's best persona. Respectable, no complaints.
\* But they cannot keep their mouth shut on clean traffic. This is the one that matters, as in the real world, almost everything on the wire is benign.
| Model | FP on benign inside malware | FP on fully benign |
|---|---|---|
| Claude Opus 4.7 | 36% | 39% |
| GPT-5.4 | 53% | 43% |
| Gemini 2.5 Pro | 41% | 86% |
| LogLM | <1% | <2% |
\* They can detect, but they can't point. Fine, it flagged a unit. Can it tell you which flows drove the call? Per-flow F1: Claude 65%, Gemini 52%, GPT 44%. LogLM sits at 99%. An alert that basically says "something in these 1,000 flows is bad, have fun" doesn't save your analyst a single minute.
\* And it's not cheap. Per single-persona alert: Claude $0.150, Gemini $0.062, GPT $0.057. LogLM is under $0.0001. Feels trivial until you do the multiplication: at a million alerts a day, even the cheapest LLM is burning \\\~$57k/day before a human looks at anything. At telco scale, you're into hundreds of millions a day, on inference alone.
Why this happens: these models have read basically everything ever written about how network traffic can be malicious, so their internal "is this flow suspicious?" prior sits way, way above the real base rate out in the wild. It stays hidden on a benchmark that's mostly malicious. The second you ask the model to sit quietly on clean traffic, it comes roaring out.
LLMs are excellent at the stuff that reads like a story with steps: triage, enumeration, chaining an exploit, turning a paragraph into a detection rule, and writing up an incident. Flow-level detection just isn't that kind of problem. There's no narrative thread to follow; the signal is buried in the distribution across thousands of connections. That's a job for an encoder, not an agent.
SOCBench is open, and I want people to poke holes in it and push it further. A benchmark for AI in security really shouldn't be one vendor's homework assignment, mine included. If you work in detection, DFIR, or hunting, I'd love a few things: datasets that look like your environment, thoughts on the scoring (especially the explainability lenses), ideas for tasks beyond detection (triage, IR, hunting, detection engineering are all next), or just someone running it and telling me where it breaks.
Repo: \[github.com/DeepTempo/socbench\](http://github.com/DeepTempo/socbench)
Full writeup with all the tables: \[deeptempo.ai/blogs/the-36-percent-false-positive-problem-with-llm-in-the-soc\](http://deeptempo.ai/blogs/the-36-percent-false-positive-problem-with-llm-in-the-soc)
Have at it in the comments.