What to log, and how long to keep it
The moment logging matters is the moment you need to answer a question about the past. When did this account first behave differently? Which machine did that connection come from? Was this file accessed before it appeared somewhere it should not have?
Either the record exists and the question takes an afternoon, or it does not and the question is unanswerable. There is no third option.
Identity events are the highest-value source, because almost every modern intrusion touches identity somewhere.
Getting logs off the machine that generates them matters more than any platform decision.
The retention problem in one sentence
At that moment, one of two things is true. Either the record exists and the question takes an afternoon, or it does not and the question is unanswerable. There is no third option, and the decision that determined which one you get was made months earlier, usually by a default setting nobody looked at.
Intrusions are frequently discovered long after they begin, and default log retention in many systems is measured in days or weeks. Put those two facts next to each other and the conclusion is uncomfortable: by the time most organisations start looking, the beginning of the story has already been overwritten.
You do not need to know exact industry averages to act on this. You need to know your own numbers. Open your identity provider, your firewall, and your endpoint tool, and write down what each one keeps and for how long. That list is usually the most useful hour anyone spends on this topic.
What to collect, in priority order
Collecting everything is expensive and, in practice, collecting everything usually means collecting nothing well. A shorter list, kept properly, beats a longer list kept for eleven days.
- Identity events. Sign-ins, both successful and failed. MFA challenges and their outcomes. Password and MFA-method changes. Privilege grants and role changes. Consent granted to applications. This is the single highest-value source, because almost every modern intrusion touches identity somewhere.
- Endpoint detection and response telemetry. Process creation, persistence mechanisms, and the alerts your endpoint tool raised and dismissed. The dismissed ones matter as much as the raised ones.
- Network edge. Firewall allow and deny decisions, VPN sessions, and DNS queries. DNS is unglamorous and frequently the fastest way to establish what a compromised machine was talking to.
- Mail. Message trace, transport rules, and any change to forwarding or delegation. Mailbox rule changes are a classic early indicator and are trivially cheap to record.
- Administrative changes. Configuration changes on your cloud tenant, your servers, and your network devices, with the account that made them. "Who changed this and when" is one of the most common questions and one of the least often answerable.
- Application and file access, for the handful of systems where the data would genuinely hurt.
How long is long enough
Retention is a business decision dressed as a technical one. Three inputs decide it.
Regulatory and contractual requirements. Some sectors and some customer contracts specify a minimum. Start here, because it is not negotiable.
Realistic detection time. How long could something plausibly run in your environment before somebody noticed? Retention shorter than that answer is retention that cannot support an investigation.
Cost. Storage is cheap; indexed, searchable storage in a logging platform is not. This is why tiering exists.
A defensible pattern for most mid-sized organisations: keep the last three to six months hot and searchable, then move to cheap cold storage for the remainder of your required period. Cold data is slower to query, but slower is not the same as gone.
Whatever you decide, write it down as a policy with a date and an owner, and check once a year that the systems actually behave the way the policy says. Retention settings drift when platforms update.
Integrity, not just existence
A log an attacker can edit is not evidence. Two properties matter:
Get the logs off the machine that generates them. Anything held only locally can be cleared by whoever gains control of that machine. Forwarding to a central collector is the single most valuable structural improvement.
Make the central copy hard to alter. Write-once storage, an append-only bucket, or a platform where deletion requires separate credentials. The same immutability principle you apply to backups applies here.
Also, synchronise your clocks. Correlating events across systems with drifting clocks is miserable, and it is entirely avoidable.
You do not need a SIEM to start
The full platform, with correlation rules and a tuned alerting pipeline, is a real project with real running costs. It is not the entry point.
The entry point is: identity logs centralised, kept for six months, and searchable by someone who knows how. That alone answers a large share of the questions organisations actually ask. Add endpoint and firewall next. Build alerting after you have data worth alerting on, not before.
Starting with the platform and then deciding what to send it is the order that produces expensive shelfware.
A test worth running
Pick a real question and try to answer it from your logs today:
- Which accounts signed in from outside your normal countries in the last ninety days?
- Has anyone created a mailbox forwarding rule in the last six months?
- Which administrator changed the firewall rule set most recently, and what did they change?
If you can answer all three in under an hour, your logging is in better shape than most. If you cannot, you have just found your priority list, and it is a shorter list than it felt like before you tried.
Questions we get about this
The ones that come up most often once this is on the table.
Does this mean we need a SIEM?
Not to start. Identity logs centralised, kept six months and searchable by someone who knows how, answers a large share of the questions organisations actually ask. Build alerting once you have data worth alerting on, not before.
How much does longer retention cost?
It depends entirely on whether the data is indexed and searchable or sitting in cheap cold storage. That is what tiering is for: keep the recent months hot and push the rest to cold. Cold is slower to query, not gone.
Does keeping logs longer create a privacy problem?
Logs contain personal data and fall under the same principles as anything else, so retention needs a stated purpose and a defined period. That is an argument for writing the policy down, not for keeping less than you can investigate with.
If we can only do one thing, what do we log?
Sign-ins including failures, MFA outcomes, and any change to a password, an MFA method or a privileged role. If one source gets centralised and retained properly, make it that one.
Where this sits with us
The services this subject falls under.
Could you answer what happened ninety days ago?
We take three real questions, try to answer them from what you record today, and hand you the gap list. Half a day, and the outcome tells you what to fix in what order.
Practical IT knowledge in your inbox
New guides on management, security and the workplace, written by the people doing the work. No sales talk, and you can unsubscribe in one click.