Skip to content

Practical steps to help be ready to investigate a compromise.

License

Notifications You must be signed in to change notification settings

jamesfed/LoggingForDisaster

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 

History

15 Commits
Β 
Β 
Β 
Β 

Repository files navigation

Logging for disaster

Practical steps to help be ready to investigate a compromise. A presentation delivered to the College IT Conference 2025.

Following on from:

As a presentation on YouTube: https://www.youtube.com/watch?v=dWX5vqk6EJ0

By James Preston of ANSecurity.

Personal blog at myworldofit.net.

By the end of this presentation you will

  • Have direction on the creation of a logging policy.
  • Understand the utility of logging in the context of incident response.
  • Be able to identify useful log sources.
  • Have options on how to obtain and store your logs.
  • Be ready to log!

My intent

  • Provide a 'quick start' point in a direction for effective logging.

What are others saying?

Center for Internet Security

National Cyber Security Centre (NCSC)

Log configuration guidance

Apache web server

From the CIS Benchmarks CIS Benchmarks

VMware ESXi

From the CIS Benchmarks CIS Benchmarks

Why do I need to log?

To answer questions, such as:

  • Sophos detected a malicious program on a computer - where did it come from?
  • Based on the data in a threat intel report have we been impacted?
  • What files have been copied to the Internet from a computer?

But also...

  • To identify the root cause of an incident (non-security related!).
  • To comply with regulations.
  • Generally a 'good idea'.

Perhaps most of all so that you don't have to use sentences like:

  • "fined for a breach that put personal information of 79,404 people at risk"
  • "The unidentified hackers"
  • "The security measures of Advanced's subsidiary fell seriously short of what we would expect from an organisation processing such a large volume of sensitive information," Mr Edwards said.
  • "There is no excuse for leaving any part of your system vulnerable," Mr Edwards added.

From: BBC News - NHS software provider fined Β£3m over data breach after ransomware attack

Beware - rabbit hole ahead!

  • Deception and honeypots.
  • Log review.

Demo 1 – data exfiltration

Validato, manual exfiltration, and Microsoft Defender EDR

Scenario setup

Scenario setup 1

Scenario setup 2

Defenders point of view

Data exfiltration-1

Data exfiltration-2

Data exfiltration-3

What should I log?

Data that'll be useful in forensic analysis

  • New application process.
  • DNS requests.
  • URL requests.
  • New/modified file and registry keys.
  • Loading of drivers.
  • Reading specific file types (e.g. .docx/.pdf/.xlsx).
  • Script/command line interface use.

Data that'll be useful in threat hunting

Abnormal events

  • AppLocker blocked the execution of a new application.
  • Excessive requests to unauthorised web or file sharing services.
  • Break glass account used... careful about to hit that rabbit hole.

Sigma rules as inspiration

Log retention

  • Minimum 90 days for all logs.
  • Target as long as you can (365 days wouldn't be abnormal) for high value logs.

How do I capture logs?

Windows

  • Deploy sysmon everywhere.
  • Increase the minimum local retention for Event Logs.
    • ~2GB limit for 'live'.
  • NXLog CE and Winlogbeat.
    • Direct out of Event Logs.
    • Read file system.

Linux

  • rsyslog.
    • imklog.
    • imjournal.
    • Read file system (e.g. for Apache).

Network appliances/Infrastructure

Cloud Services

  • API connectors.

From LimaCharlie. LimaCharlie

Where should those logs go?

Consider: authentication and encryption

  • Network appliances don't always support both/either.
  • May need to use some form of logging 'proxy'.
  • If running a 'cloud' logging service the importance is even greater.
  • Don't underestimate a threat actors resourcefulness, easy to generate sufficient logs to overwhelm a system or inject malicious content into logs.

If storing on-premises consider

  • Network level access controls to the log system.
  • Authentication to the log system.
  • Dependencies on systems that may be offline in an incident (compute, storage, networking, authentication).

A recommendation for on-premises

  • Place the server(s) in a dedicated 'logging' network.
  • Restrict access into that network with a network firewall, broadly allow connections to the ports that logging agents talk to, restrict by source network/IP/user/device access to the management interface.
  • Dedicated physical server or servers (3 smaller nodes in a cluster is better than a single node by itself).
  • Strong authentication (phishing resistant) as the 'day to day' access, with a fallback to local authentication with a strong (long) passphrase.

Volume vs value (examples)

  • Firewall traffic logs - VERY high volume, low relative value.
  • DNS logs - high volume, low-medium relative value.
    • But.... DNS is encrypted now.
  • Command line logs - low-medium volume, high relative value.
  • Previously unseen application launched (or attempted to) - low volume, high relative value.

Sharing your logs

  • Volume (TBs of logs!) is often the main problem.
  • Bulk file transfer of text files.
  • Not uncommon to grant read-only access to the centralised log store.
  • Export from >X<Search to JSON.
  • Export with backup feature and then import.

Demo 2 – Malware detection triggers investigation

Validato, Microsoft Defender, LimaCharlie EDR

Detection by Microsoft Defender

Defender awakes!

Defenders point of view

EDR-1

EDR-2

EDR-3

EDR-4

EDR-5

EDR-6

Demo 3 – Web shell party mode πŸŽ‰πŸ₯³

VirusTotal, wordpress_shell.php, LimaCharlie EDR.

Web shell detected!

WordPress webshell

Threat actors point of view

ifconfig

Or coninminer

Defenders point of view

INVESTIGATE!

INVESTIGATE!!

INVESTIGATE!!!

Some other cool things you can do with logs

Nessus

Be amazed by the Christmas spike in brute force attacks

Get longer data retention than cloud services provide

When you get back to your institutions

  • Talk with senior leadership about establishing a formal policy for logging.
    • Then take that policy and turn it into practice/procedure.
  • Find 1 (or 3!) old(er) servers (desktops), fill them with disks (8TB SSDs are cheap), configure a logging stack of your choice.
    • Get firewall, web server, AV agent, and authentication logs flowing into them.
  • Review options for an EDR service.
    • LimaCharlie really is a great entry point, try it out on some web servers and go snooping!

But also don't forget to:

  • Do maintenance! Don't let that logging system run out of disk space!
  • Check all log sources are sending logs.
  • Routine check that the times of systems are in sync.

If you have the spare time:

Logging and being able to respond is great...

But a good defence is still a good defence.

  • Build a culture of reporting abnormal events.
  • Install AV everywhere, including on Linux web servers!
  • MFA (or even better strong authentication) everything!
  • Decrypt!
    • Inbound (where you own the private key already).
    • Outbound (where you generate new trusted certificates on the fly for services that you don't have the private key for).
  • Configure outbound filtering!
    • Block access to remote access tools (legitimate and otherwise).
    • Strict outbound filtering from services that allow connections from untrusted networks/devices.
  • Test your backups!

ANSecurity

WAF

About

Practical steps to help be ready to investigate a compromise.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published