Security Data Lake vs SIEM: When to Use Each (and When to Use Both)

Every year around SIEM renewal time, a version of the same conversation happens in a lot of SOCs. The quote went up again, and somebody suggests moving some of the data to a data lake. It's a reasonable idea, but it's also vague enough to mean almost anything, from "archive it in S3 and hope we never need it" to a real change in how the team stores and searches its data.

For most security teams, the short answer is that you shouldn't pick one. A SIEM and a security data lake are good at different jobs, and the teams getting the most out of both have stopped treating it as an either-or decision. The hard part isn't choosing a tool. It's deciding which data goes where, and who makes that call.

This post walks through what each system does well, where each one breaks, and how to tell which your environment needs.

What a SIEM is good at

A SIEM is where detection happens. It takes in security data, runs correlation rules against it in close to real time, raises alerts, and gives analysts one place to triage and investigate. For most SOCs it also holds years of accumulated work: tuned detection rules, saved searches, dashboards, and the muscle memory of every analyst who knows the query language.

That's hard to replace, and you usually shouldn't try.

Where a SIEM struggles is the economics of keeping data. Most SIEMs price by how much data you send them, measured in gigabytes per day. That model works fine for the curated slice of data your detections run against. It becomes painful for everything else: the firewall logs, DNS queries, and cloud audit trails you might need during an investigation but won't alert on every day.

So teams make trade-offs. Hydrolix estimates that most organizations send roughly 30% of their data to the SIEM and archive or drop the other 70%. Retention windows shrink to 7 or 30 days, set by what the license covers rather than what an investigation needs. Hydrolix's Michael Cucchi calls this "security by budget," which is about as accurate a description as exists.

The risk shows up later. Scanner points out that median attacker dwell time for cyber espionage runs from 122 to 400 days, while most teams keep 30 to 90 days of searchable logs. When an incident finally surfaces, the evidence of how it started has often already aged out.

What is a security data lake?

A security data lake is a place to keep large volumes of security data cheaply, in an open format, for as long as you need it. Instead of paying a SIEM's price per gigabyte, you store data in something like Amazon S3 at a fraction of the cost and query it when you need to.

The classic trade-off is speed. A traditional data lake is cheap because it's built for storage, not for fast searching. Querying months of raw logs in a basic lake can take minutes or hours and often needs someone who knows SQL and how the data is laid out. That's fine for a quarterly compliance report. It's not fine for an analyst in the middle of an incident.

That gap is where a newer group of tools has shown up, and it's worth knowing the difference, because "data lake" now covers a wide range of options:

Cold storage lakes (S3, or open table formats like Apache Iceberg and Delta Lake) are the cheapest option and the slowest to query. They're good for compliance retention and archives.

Hot, queryable data layers like Hydrolix keep full-fidelity data compressed and searchable for months instead of days. Hydrolix cites 15 to 24 months of hot retention, and it can be queried from inside Splunk through a connector, so analysts don't have to learn a new tool. In testing with RAD on Palo Alto firewall data, broad correlation searches over the extended dataset returned in about 2 seconds.

Search on your own lake, like Scanner, indexes the logs sitting in your S3 buckets so they can be searched quickly without moving them into a SIEM. Scanner's customer Ramp stretched its searchable history from 15 days to one year and cut query times from more than 30 minutes to about one minute.

The point for a buyer: "move it to a data lake" can mean anything from "cheap and slow" to "cheap and fast enough to investigate with." Ask which one someone means.

Side by side: SIEM vs Data Lakes

When a SIEM alone is enough

Plenty of teams don't need a data lake yet. If your data volume is modest, your compliance windows are short, and the SIEM bill isn't forcing you to drop sources, adding a second system brings in more to manage than it saves. The time to revisit is when you notice you're cutting sources or shortening retention to stay under budget.

When you need a data lake

The signals tend to look like this. Investigations stall because the data aged out. Compliance requires you to keep a year or more of logs. Your threat hunters want to look back further than the SIEM allows. Data volume is growing faster than the security budget. Or you're building AI agents and analytics workloads that need to query large amounts of history without paying SIEM prices for every search.

Scanner's research on agentic security operations adds one more: AI agents are only as good as the data they can reach. An agent that can only see 30 days of logs, spread across systems that can't be searched together, will give you shallow answers no matter how capable the model is.

When you need both, and what connects them

For most mid-size and enterprise SOCs, the answer is both. Detection stays in the SIEM, where your rules and analysts already live. Full-fidelity retention moves to a lake or a hot data layer built for it. We call this decoupling retention from detection.

The piece that makes it work is the layer in between. Something has to collect data from every source, put it in a consistent format, and decide where each event goes: the high-value signal to the SIEM, a full copy to Hydrolix or your Scanner-indexed lake, and compliance archives to cold storage. Without that layer, every new destination means rebuilding collection, and the split ends up decided by whichever integration was easiest to set up.

That's the job of a security data pipeline. Monad collects from 330+ sources, normalizes and enriches the data, and routes it to 40+ destinations, so each event lands where its value justifies the cost. Two examples: in the RAD test, Monad routed the firewall data into Hydrolix while the existing Splunk feed stayed untouched, and at Lambda, Monad and Scanner together widened visibility and set up agentic detection.

Questions to ask before you buy

• Can my analysts query it from the tools they already use, or do they need to learn a new interface?
• What happens to my existing detections? Do any of them search further back than my new SIEM retention window?
• How fast are queries over six months of data, measured on my data, not a demo dataset?
• Who owns the data format? Can I leave without re-ingesting everything?
• What sends data to it, and what happens to that feed when a source changes its API?

Where to start

Pick the one source you've been trimming for cost, usually firewall, DNS, or EDR data. Price out what it costs in your SIEM today versus a data lake for the retention window you actually want. Monad's storage cost analysis tool will run that comparison for you, and if you want help mapping your sources, reach out at product@monad.com.

‍