Security monitoring explained: IDS, SIEM, EDR and where the logs go

Security monitoring is the practice of watching computers and networks closely enough to notice when something harmful is happening, ideally early enough to stop it. The field is full of three letter abbreviations, and vendors use them loosely, but underneath there are only a few families of tools. Each family watches a different place, and the logs they produce flow to a central store where a person or a rule can look at them together. This article gives you the map.

An everyday analogy

Think of a building with a security team. Cameras at the gates record everyone who comes and goes; that is network monitoring. Motion sensors and door contacts inside each room notice unusual movement; that is endpoint monitoring. All the feeds and sensor signals arrive in a control room where a guard watches screens and an alarm panel highlights anything that matches a known pattern; that is the SIEM. Occasionally the team pays someone to try to sneak in, so they can check that the cameras and sensors really do raise the alarm; that is detection testing. No single part is enough on its own. Cameras miss what happens behind a closed door, room sensors cannot see who walked in from the street, and both are useless if nobody is watching the control room.

The families

Network intrusion detection (NIDS) looks at a copy of the traffic flowing through a network and raises an alert when a packet or a conversation matches a signature of a known attack. When the same tool sits inline and can drop traffic, it is called an intrusion prevention system (IPS). Network security monitoring (NSM) is a close cousin that cares less about alerts and more about writing a complete, compact record of what crossed the wire, so analysts can reconstruct events later.

Host intrusion detection (HIDS) runs as an agent on each server or workstation and watches what happens inside it: logins, process starts, file changes, configuration drift. Endpoint detection and response (EDR) is the modern, deeper version of the same idea: it records process behaviour in detail, correlates it with attacker techniques and lets a responder isolate the machine or kill a process remotely. Runtime security for containers applies the same thinking to Kubernetes, where the unit of interest is a container rather than a whole host.

Security information and event management (SIEM) is the control room. It collects logs from every source above, plus firewalls, web servers, identity systems and cloud accounts, stores them so they can be searched, and runs detection rules across all of them. XDR, extended detection and response, is a marketing term for a product that combines EDR, network data and SIEM style correlation in one package. Intrusion prevention based on logs, in the style of fail2ban, is a small but useful corner: read a log, spot a hostile address, block it at the firewall.

Where the logs go

The flow is nearly always the same. Sensors and agents generate events. A shipper on each machine forwards them over the network to a central collector. The collector normalises the fields, stores the events in a searchable index and keeps them for a retention period dictated by disk space and regulation. Detection rules, often written in the vendor neutral Sigma format, run on a schedule against the index and create alerts. Alerts go to a queue or a case management system where an analyst decides whether they are real. Everything is kept so that when something is confirmed, the analyst can go back through weeks of logs from every source and build the full timeline. The Monitoring topic of this encyclopedia covers the shipping and storage side in detail; the security tools simply plug into the same pipeline.

Where each open source project sits

• Suricata is a network IDS and IPS: it matches traffic against signatures and can drop packets inline.
• Zeek is network security monitoring: it writes detailed protocol logs of every connection for later analysis rather than alerting on signatures.
• Wazuh is a host intrusion detection system and small SIEM in one, with agents that watch logs, files and packages and a central dashboard that correlates them.
• Falco is runtime security for Linux and Kubernetes: it watches system calls and alerts on suspicious behaviour inside containers.
• osquery is an endpoint visibility agent: it exposes machine state as SQL tables so you can ask every host the same question and record how answers change.
• CrowdSec is log based intrusion prevention: it detects hostile addresses in logs, blocks them, and shares what it learned with a community blocklist.
• pySigma is detection rule tooling: it converts Sigma rules into the query language of whichever SIEM you run.
• Atomic Red Team is detection testing: it lets you safely imitate attacker techniques on a lab machine to see whether the tools above notice.
• Security Onion is the whole building at once: a Linux distribution that bundles Suricata, Zeek, Elastic and investigation tools into one installable platform.

Where to start

If you administer a handful of servers, start by shipping their logs somewhere central and adding CrowdSec or fail2ban; that already stops most noise. The next step is a host agent such as Wazuh, which gives you file integrity and vulnerability reports as well as alerts. Add a network sensor only when you have a place to put it and someone to read its output; Security Onion is the easiest way to try one. Whatever you deploy, run a few Atomic Red Team tests in a lab afterwards so you know the alerts really arrive. A quick check that a tool is installed is usually its version flag:

suricata --version
zeek --version
falco --version
osqueryi --version

Official documentation

• Suricata user guide
• Wazuh documentation
• Security Onion documentation