It was a quiet Friday when an operations engineer decided to open the folder everyone avoided: the place where the team’s detection rules lived. Nothing was on fire. She simply wanted to know whether the rules, the ones that are meant to warn about suspicious activity on company servers, were in a state she could defend to an auditor. By evening she had a list of twenty-odd questions, and a clearer sense of what DetectionOps asks of a team. The list below is that audit, tidied up so you can run it yourself, with the reasoning from the Ops Atlas entry attached to each block. The engineer is invented, but the questions are real.
Before you start: what you are auditing
DetectionOps, as the Ops Atlas reference page for DetectionOps lays it out, is the management of detection logic: the rules themselves, the requirements behind them, tests, coverage evidence and the tuning and retirement history. The audit therefore ignores the alert queue, the hunting and the response drills, which sit with other roles. Keep the scope that tight or the checklist will sprawl.
Gather three things first: an export of every rule, the list of exclusions or suppressions attached to them, and the log of recent changes. If any of the three does not exist, that is your first finding.
Block 1: Where does the truth live?
- Is every rule stored in version control, with an author and a date on each change?
- Does production match the repository exactly, or has someone edited rules in the console?
- Is there an automatic check, even a simple linter, on each proposed change?
The Atlas calls the state where the first two answers are yes “versioned,” and it is level one on its five-step ladder. The engineer found that eleven of her rules differed from the repository copy, which told her more than any dashboard.
Block 2: Does each rule know why it exists?
- Can you point to a written requirement for each rule, naming the behaviour it targets?
- Are blind spots and known innocent look-alikes recorded?
- Do rules carry one of the five lifecycle labels the Sigma format defines (the list runs from experimental up to stable, then deprecated and unsupported), and is anyone actually retiring the deprecated ones?
Those status words come from the Sigma specification that the entry covers. A repository full of rules flagged stable but untouched for years is a smell, because stability should be earned through measurement.
Block 3: Are the tests honest?
- Does each rule have one test that should fire it and one that should not?
- Were those tests written from the intended behaviour rather than copied from the rule’s output?
- Do accepted blind spots appear as tests of their own, with a responsible person and a lapse date?
The Atlas defines test completeness as the share of rules with tests, and shows 84 of 120 as an example score of seventy per cent. Level two on its ladder asks for ninety per cent or better.
Block 4: Do the rules depend on data that exists?
This block surprised her most. The entry gives a concrete case: a rule for newly created scheduled tasks relies on a particular Windows event, number 4698, and that event only appears if one specific audit subcategory is turned on. It adds that direct registry writes, which need the highest privilege level, can sidestep the logging that the rule listens to. The Atlas maps that technique, T1053.005, to a detection strategy identified as DET0441 in the current ATT&CK release. The point for the checklist is plain: for each important rule, confirm the log source is switched on and that you know which evasions would slip past it.
- Is the required log source enabled on every host the rule is meant to watch?
- Do you know at least one way an attacker could avoid generating the evidence?
Block 5: How do changes get out?
- Is there a trial period where new rules run silently, roughly a week, before they can page anyone?
- Is there a written rule for urgent changes? The Atlas’s proposal lets an emergency change skip staging, never tests or review, and be reviewed again within five working days.
- Do exclusions require analyst evidence and an expiry date?
Our related reading on managing delivery risk reliably and securely sets out why such guardrails protect speed rather than hinder it.
Block 6: What do the numbers say?
- What is the precision of your five noisiest rules? The Atlas works one through: a rule raising forty alerts of which six were real scores just fifteen per cent.
- How many analyst minutes does each genuine finding cost? The example rule consumed about forty.
- When did you last validate coverage by emulating attacks, instead of counting tags? The entry’s sample shows 60 per cent validated against a tag heatmap that claimed everything.
- What share of rules are healthy, and how many deployments in a period needed a rollback?
Block 7: Is the toolchain frozen?
- Are the versions of every conversion tool, backend and library pinned?
- Is there a planned upgrade day rather than surprise upgrades?
- Does the platform you deploy to have a published retirement date? The Atlas flags a dated example for one cloud SIEM. For that product, managing rules in the Azure portal is slated to lose support after 31 March 2027, with Microsoft steering new rules toward a custom-detections feature that remains in preview and needs a premium licence; whatever your platform, retirement notices like this one deserve a calendar entry.
Scoring and what she did next
Count the yes answers per block, then place yourself on the ladder: ad hoc, versioned, tested, validated, or a managed portfolio. The engineer landed between versioned and tested and wrote a ninety-day plan that mirrors the entry’s outline: Git and lint in the first month, tests for the noisiest twenty rules in the second, and a first emulation exercise in the third. She also held back from repairing everything in one sweep, and followed the Atlas’s suggestion to start with the threats that matter most to the business.
If the vocabulary felt new, our beginner glossary of fifty-plus delivery terms will fill in the standard words, and the audit itself needs no more than a spreadsheet and an afternoon. Her last note on the page was short: write down the date of the audit, because next quarter’s version will be a comparison, and comparisons are what turn a checklist into a practice.









