Two engineers at an imaginary company in Osaka once argued over lunch about the same question for a full hour: where should the firewall logs live? One wanted everything in the search platform, so any analyst could query it instantly. The other wanted raw files in cheap storage and only summaries upstairs. They were both partly right, and the argument is a perfect doorway into SecDataOps, the practice of managing security logs as evidence with owners, contracts and a budget. The sections below put the usual choices next to each other and show what each one costs and where it cracks.
Setting the scene
The Ops Atlas describes the discipline in an entry you can read as the Atlas reference on SecDataOps. In its words the job is to onboard, shape, enrich, route, tier and retain security telemetry so that detections, investigations and audits have evidence they can lean on. I will compare three ways teams meet that challenge, and then add a fourth that blends the strengths of the others.
Approach A: everything in the search platform
What it feels like. Simple. One product, one query language, one bill. New log sources are a checkbox away, and analysts love the instant answers.
Where it hurts. The price grows with volume, and volume grows by itself. Teams respond by trimming sources when the invoice arrives, usually without a map of which rules need which fields. The Atlas flags a related misunderstanding: people assume the licence covers log management, when in practice such a product searches whatever reaches it and says nothing about what failed to arrive.
Best for. Small environments with modest volume and a short, well-understood list of sources.
Approach B: raw lake first, search later
What it feels like. Cheap, patient and a little slow. Everything is written once to object storage, perhaps in a columnar format, and queried when needed.
Where it hurts. Without alerting, a lake is a library, not a guard. The Atlas notes cloud services whose cheapest plans offer minimal ingest cost but slow queries and no alerts at all. It also notes that one major provider's security lake did not offer object-lock protection as of October 2026, which matters if you need tamper resistance.
Best for. Long-term retention, compliance archives and investigations that can wait minutes for an answer.
Approach C: tiered pipeline with contracts
What it feels like. More work up front, calmer afterwards. A pipeline sits in the middle, parsing events, trimming noise, redacting sensitive values and routing each stream to the right tier. Every source has a one-page contract.
Where it hurts. It needs owners. Someone has to maintain parsers as code, run the tests and watch health. Beginners also stumble on tool quirks, such as the Atlas's warning that a conversion in one popular forwarder can turn a null into zero.
Best for. Mid-sized and large organisations whose bills or audit duties have outgrown approach A.
A side-by-side view
- Speed of first answer: A is fastest, C is close for hot data, B is slowest.
- Cost at scale: A is highest, B lowest, C in between and tunable.
- Retention depth: A is shallow by necessity, B and C can go very deep.
- Alerting: A and C have it natively, B usually needs add-ons.
- Operational effort: A is low, B medium, C highest until the routines settle.
- Resilience to source drift: C wins if the contract tests are real.
The numbers that settled the lunch argument
Back to Osaka. The two engineers finally sketched the Atlas's worked example on a napkin. Take a firewall that emits four hundred gigabytes daily: dropping all of it into the hot tier ran to roughly 400 cost units a day. Holding only eight gigabytes of digests hot while the raw feed went into a lake that keeps it 400 days came to about 164, while raw retention rose from 90 to 400 days. The napkin made the point more gracefully than the argument had: the answer was a mix, with approach C handling the details.
The comparison also teaches a quiet lesson about timelines. Many Japanese teams are in the middle of growth in cloud usage, which our forecast for DevOps practice in Japan and beyond covers, and every new service adds a log source. Deciding the storage policy early is much cheaper than retrofitting it.
Retention is where comparisons turn serious
Each approach implies a default retention, and defaults are dangerous. Public rules set different floors: an American federal memo wants twelve months readily searchable plus eighteen more in cold storage, Australian-led guidance starts at a year for admin and security logs, and an Indian directive from 2022 sets 180 days. The Atlas also warns about a billing quirk in one cloud monitoring service where the first thirty-one days are already included, so shortening retention below that line saves nothing at all.
The risk viewpoint is useful here. As described in our discussion of weighing reliability and security risk in delivery work, a retention decision is a risk decision. You are choosing how much history you can afford to be without.
What each approach means for the people involved
Beyond cost and speed, the choice changes daily life for each role. Analysts in approach A enjoy the quickest answers but spend energy begging for budget to add sources. Auditors prefer approach B or C, since deep, ordered archives with documented retention make their sampling easier. Platform engineers carry the heaviest load in approach C, because they maintain parsers, tests and monitors, but they also gain the most control. Finance sees the biggest swing: a tiered design makes spend predictable per stream, while a search-first design makes it a surprise every quarter.
There is a human side to ownership too. In approach A the logs often belong to everyone, which means they belong to no one. Approach C forces a conversation about who is responsible for each link, and that conversation is frequently the most valuable output of the entire exercise.
A migration story, kept deliberately small
Return to our imaginary Osaka company. Their move started with a single stream: office network flows, the loudest and least-read data they had. They wrote a contract for it, set a freshness target, added a canary, and began writing raw copies to the archive in parallel with the old path. For two weeks both paths ran. Then they replayed a week of raw data against the summary path, confirmed every rule still triggered, and cut the old route. Nothing dramatic happened, which is exactly how a good migration should feel.
The fourth option: use all three on purpose
The mature answer is rarely a single approach. The Atlas describes tiers that cooperate: a small hot tier for daily detection, a warm tier for investigations, a deep archive for raw originals, and a pipeline that keeps them consistent. The raw archive acts as the safety net, because normalised copies can be regenerated but originals cannot.
To decide how to move, consider the stages the Atlas lays out, from reactive to inventoried, contracted, measured and finally optimising. Teams in approach A are usually reactive. Teams who have built a register and contracts are somewhere between approaches B and C.
How to choose without a fight
- List your ten most important sources and their daily volumes.
- For each, write down the rules or audits that read it and the fields they use.
- Price the stream in hot storage and in a mixed arrangement.
- Replay a week of raw data through any proposed change and confirm every dependent rule still fires.
- Run a silence test to prove you would notice if the feed stopped.
The two engineers in Osaka did exactly that over the following month. The funny part is that they ended up agreeing on almost everything but the name of the new bucket. It is a small, human reminder that the best comparison is not a contest between tools but a conversation between the people who own each link in the chain. Write the agreement down, test it, and the argument turns into a plan.









