Build a monitoring stack with Prometheus, node_exporter and Grafana on one Linux server

This tutorial builds a complete, small monitoring stack on a single Linux server. Prometheus collects metrics, node_exporter exposes the server’s CPU, memory, disk and network numbers, and Grafana turns them into a dashboard and sends an alert when something is wrong.

It is aimed at administrators who know what a systemd service is and want to see how the pieces fit together before monitoring a whole fleet. Everything runs on one machine, so you can build it in a virtual machine and throw it away afterwards.

What you will build

• Prometheus and node_exporter installed from the official release archives, each running as its own system user under systemd.
• A scrape configuration with two jobs (Prometheus itself and the node), a recording rule, and a safe way to reload it.
• A set of PromQL queries for CPU, memory, disk and network that you can reuse.
• Grafana OSS installed from the official APT repository, with Prometheus as a data source.
• A dashboard with four panels and an alert rule for high CPU that notifies an email contact point.

Before you start

• A server: Debian or Ubuntu on x86-64 (amd64), at least 2 CPU cores, 2 GB RAM and a few GB of free disk. A virtual machine is ideal. Commands use the placeholder name mon1.example.com (192.0.2.30).
• Access: SSH and a user with sudo, plus a browser that can reach port 3000 on the server (or an SSH tunnel).
• For email alerts: an SMTP server you are allowed to send through. Without it you can still build the alert and use a webhook contact point instead.
• Time: about 75 minutes.
• Knowledge: systemd basics (systemctl, journalctl), editing files as root, and the Prometheus idea of pulling metrics from exporters. A Prometheus concepts article is a good warm-up.

Step 1: Create system users and directories

Each service runs as its own unprivileged user without a login shell. If one is compromised, it cannot touch the other’s files.

sudo useradd --system --no-create-home --shell /usr/sbin/nologin prometheus
sudo useradd --system --no-create-home --shell /usr/sbin/nologin node_exporter
sudo mkdir -p /etc/prometheus/rules /var/lib/prometheus
sudo chown -R prometheus:prometheus /var/lib/prometheus

Check with id prometheus and id node_exporter; both should print a user and group.

Step 2: Download and install the binaries

The Prometheus project publishes precompiled binaries for each release, together with a sha256sums.txt file. Pick the current versions on the download page; 3.15.0 and 1.12.1 were current when this was written.

cd /tmp
PROM=3.15.0
NODE=1.12.1
wget https://github.com/prometheus/prometheus/releases/download/v$PROM/prometheus-$PROM.linux-amd64.tar.gz
wget -O prom-sha256sums.txt https://github.com/prometheus/prometheus/releases/download/v$PROM/sha256sums.txt
wget https://github.com/prometheus/node_exporter/releases/download/v$NODE/node_exporter-$NODE.linux-amd64.tar.gz
wget -O node-sha256sums.txt https://github.com/prometheus/node_exporter/releases/download/v$NODE/sha256sums.txt
sha256sum -c --ignore-missing prom-sha256sums.txt
sha256sum -c --ignore-missing node-sha256sums.txt

Both checks must print OK for the archive you downloaded. Do not continue if they do not. Then unpack and install:

tar xzf prometheus-$PROM.linux-amd64.tar.gz
tar xzf node_exporter-$NODE.linux-amd64.tar.gz
sudo install -m 0755 prometheus-$PROM.linux-amd64/prometheus prometheus-$PROM.linux-amd64/promtool /usr/local/bin/
sudo install -m 0755 node_exporter-$NODE.linux-amd64/node_exporter /usr/local/bin/
prometheus --version
node_exporter --version

Step 3: Run node_exporter as a service

node_exporter publishes host metrics on port 9100 at /metrics. It has no login by default, so on a single server bind it to 127.0.0.1: only the local Prometheus needs it. Create /etc/systemd/system/node_exporter.service:

[Unit]
Description=Prometheus Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
ExecStart=/usr/local/bin/node_exporter --web.listen-address=127.0.0.1:9100
Restart=on-failure

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter
systemctl status node_exporter --no-pager
curl -s http://127.0.0.1:9100/metrics | grep -E '^node_(load1|memory_MemAvailable_bytes)'

The service should be active (running), and the last command prints two lines with numbers, the 1-minute load average and the available memory in bytes.

Step 4: Configure Prometheus

The configuration says how often to scrape, which targets to scrape and which rule files to load. Create /etc/prometheus/prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "rules/*.yml"

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node"
    static_configs:
      - targets: ["localhost:9100"]
        labels:
          env: "lab"

The extra env label is attached to every series from that target, which is handy once you have several environments. Add a recording rule in /etc/prometheus/rules/node.yml. It precomputes the busy CPU share per instance, so dashboards and alerts can use a short, fast name:

groups:
  - name: node
    rules:
      - record: instance:node_cpu_busy:ratio
        expr: 1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))

Validate both files before starting anything. promtool ships in the same archive:

sudo chown -R prometheus:prometheus /etc/prometheus
promtool check config /etc/prometheus/prometheus.yml

Checking /etc/prometheus/prometheus.yml
  SUCCESS: 1 rule files found
 SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax

Checking /etc/prometheus/rules/node.yml
  SUCCESS: 1 rules found

Step 5: Run Prometheus as a service

Create /etc/systemd/system/prometheus.service. ExecReload sends SIGHUP, which makes Prometheus re-read its configuration and rule files without a restart; a broken file is not applied.

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=15d \
  --web.listen-address=:9090
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now prometheus
systemctl status prometheus --no-pager
curl -s http://localhost:9090/-/ready

The last command prints Prometheus Server is Ready. If the service keeps restarting, read journalctl -u prometheus -n 50; the usual cause is a permission problem on /var/lib/prometheus or a YAML error.

Prometheus has no login either. If you do not want it reachable from the network, change the listen address to 127.0.0.1:9090 and use an SSH tunnel such as ssh -L 9090:localhost:9090 [email protected] to open the web UI.

Step 6: Verify the targets

Open http://192.0.2.30:9090 and go to Status, then Target health (called Targets in older versions). Both jobs should be UP. The same information is available from the HTTP API:

curl -s http://localhost:9090/api/v1/targets | python3 -c 'import json,sys; [print(t["labels"]["job"], t["health"], t["lastError"]) for t in json.load(sys.stdin)["data"]["activeTargets"]]'

node up
prometheus up

A target that is down shows its last error, for example a connection refused when node_exporter is not running.

Step 7: Query the metrics with PromQL

Use the query page of the Prometheus UI, or promtool query instant http://localhost:9090 'QUERY' on the command line. Try these one by one; each answers a common question.

up

node_load1

count without (cpu, mode) (node_cpu_seconds_total{mode="idle"})

100 * (1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])))

100 * node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes

100 * node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}

rate(node_network_receive_bytes_total{device!="lo"}[5m])

instance:node_cpu_busy:ratio

• up is 1 for every target that was scraped successfully and 0 otherwise. up == 0 lists only the broken ones.
• The count query returns the number of CPU cores.
• node_cpu_seconds_total is a counter, so it is wrapped in rate() over a time window. The result is CPU busy percent averaged over all cores.
• Memory and disk queries return the percentage still available.
• The network query shows bytes per second per interface. != excludes the loopback device; =~ and !~ do the same with regular expressions.
• The last query reads the recording rule from Step 4. It may take one evaluation interval to appear.

Never use rate() on a gauge such as node_load1; it only makes sense for counters, which by convention end in _total.

Step 8: Install Grafana

Grafana Labs runs an APT repository. Add its signing key and the stable repository, then install Grafana OSS:

sudo apt-get install -y apt-transport-https wget gnupg
sudo mkdir -p /etc/apt/keyrings
sudo wget -O /etc/apt/keyrings/grafana.asc https://apt.grafana.com/gpg-full.key
sudo chmod 644 /etc/apt/keyrings/grafana.asc
echo "deb [signed-by=/etc/apt/keyrings/grafana.asc] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install grafana
sudo systemctl daemon-reload
sudo systemctl enable --now grafana-server
systemctl status grafana-server --no-pager

Grafana listens on port 3000. Open http://192.0.2.30:3000 and log in with the default user admin and password admin; Grafana asks you to set a new password right away. Do that before the port is reachable from any untrusted network.

Step 9: Add Prometheus as a data source

1. In the left menu click Connections, then Add new connection.
2. Search for Prometheus, select it and click Add new data source.
3. Set Prometheus server URL to http://localhost:9090. Grafana runs on the same host, so localhost is right here.
4. Scroll down and click Save & test.

You should see Successfully queried the Prometheus API. If Grafana ran in a container instead, localhost would point to the container itself and you would need the host’s address.

If you prefer files to clicks, the same data source can be provisioned with a YAML file in /etc/grafana/provisioning/datasources/; a provisioned data source cannot be edited in the UI afterwards.

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://localhost:9090
    jsonData:
      timeInterval: 15s

Step 10: Build a dashboard

Create a new dashboard from Dashboards, New, New dashboard, and add a visualization. Choose the Prometheus data source, switch the query editor to Code mode and paste a query. Build four panels:

• CPU busy %, Time series: 100 * instance:node_cpu_busy:ratio, unit Percent (0-100).
• Memory available %, Gauge: 100 * node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes.
• Root disk free %, Stat: 100 * node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}.
• Network in, Time series: rate(node_network_receive_bytes_total{device!="lo"}[5m]), unit bytes/sec, legend {{device}}.

Save the dashboard with a name such as mon1 host. Set the time range at the top right to the last hour; the panels should show lines, not No data. Button labels move between Grafana versions, but the flow is always data source, query, visualization type, save.

Step 11: Add a contact point and an alert rule

Grafana sends notifications through contact points. For email, Grafana OSS needs SMTP settings first. Edit the [smtp] section of /etc/grafana/grafana.ini and restart:

[smtp]
enabled = true
host = smtp.example.com:587
user = [email protected]
password = CHANGE_ME
from_address = [email protected]
from_name = Grafana

sudo chmod 640 /etc/grafana/grafana.ini
sudo systemctl restart grafana-server

Create the contact point:

1. Go to Alerts & IRM, Alerting, Notification configuration (Contact points in older versions) and open the Contact points tab.
2. Click + New contact point, name it ops-email, choose the Email integration and enter an address such as [email protected].
3. Click Test and send a test notification, then save.

Now the alert rule, from Alerts & IRM, Alert rules, + New alert rule:

1. Name: High CPU on mon1.
2. Query: data source Prometheus, query 100 * instance:node_cpu_busy:ratio; set the condition to fire when the value is above 80. Click Preview to see the current value.
3. Folder and labels: create a folder such as Infrastructure and add the label severity=warning.
4. Evaluation: create an evaluation group with a 1m interval and set the pending period to 5m, so a short spike does not page anyone.
5. Notifications: select the ops-email contact point.
6. Message: add a summary such as CPU above 80% for 5 minutes on {{ $labels.instance }}, then click Save rule.

The rule shows as Normal in the list. It moves to Pending as soon as the condition is true and to Firing after the pending period.

Check that it works

Create real load and watch the whole chain react. On the server, start one busy loop per core for ten minutes:

for i in $(seq "$(nproc)"); do timeout 600 sh -c 'while :; do :; done' & done
wait

• Within a minute, the CPU panel on the dashboard climbs towards 100%.
• In Prometheus, instance:node_cpu_busy:ratio returns a value close to 1.
• The alert rule goes to Pending, then to Firing after five minutes, and the email arrives.
• When the loops end, the rule returns to Normal and a resolved message is sent.
• curl -s http://localhost:9090/api/v1/targets still shows both targets up.

If something goes wrong

• Prometheus does not start. Run promtool check config /etc/prometheus/prometheus.yml and journalctl -u prometheus -n 50. YAML indentation errors and a data directory not owned by prometheus are the most common causes.
• The node target is down with connection refused. node_exporter is not running or listens on another address. Check systemctl status node_exporter and that the target in prometheus.yml matches the listen address.
• A configuration change has no effect. Prometheus needs a reload: sudo systemctl reload prometheus. A broken file is rejected and the old configuration keeps running, so check the journal after every reload.
• Grafana panels say No data. Run the same query in the Prometheus UI. If it returns nothing there, the problem is the query or the scrape; if it works there, check the data source URL and the dashboard time range.
• No email arrives. Use the contact point’s Test button and read journalctl -u grafana-server for SMTP errors. Email notifications in Grafana OSS do not work until [smtp] is enabled.

Clean up

Warning: the commands below delete all collected metrics, the Grafana database with your dashboards and alert rules, and the configuration. Export dashboards you want to keep first (dashboard settings, JSON model) and copy /etc/prometheus somewhere safe.

sudo systemctl disable --now grafana-server prometheus node_exporter
sudo apt-get remove grafana
sudo rm /etc/apt/sources.list.d/grafana.list /etc/apt/keyrings/grafana.asc
sudo rm /etc/systemd/system/prometheus.service /etc/systemd/system/node_exporter.service
sudo systemctl daemon-reload
sudo rm /usr/local/bin/prometheus /usr/local/bin/promtool /usr/local/bin/node_exporter
sudo rm -r /etc/prometheus /var/lib/prometheus
sudo userdel prometheus
sudo userdel node_exporter

apt-get remove keeps Grafana’s configuration and data in /etc/grafana and /var/lib/grafana; delete those directories too if you want nothing left.

Next steps

• Add more servers: install node_exporter on each, bind it to an address Prometheus can reach, protect port 9100 with a firewall and list the targets in a new job, or use file-based service discovery.
• Add Alertmanager and write alert rules in Prometheus itself, so alerts work even when Grafana is down.
• Watch services, not only hosts: the blackbox exporter probes HTTP, TCP and DNS endpoints from the outside.
• Keep the configuration in Git and deploy it with Ansible, including promtool check config as a validation step.
• Plan retention and disk size: the --storage.tsdb.retention.time flag controls how long data is kept.

Official documentation

• Monitoring Linux host metrics with the Node Exporter
• Prometheus: Getting started
• Install Grafana on Debian or Ubuntu
• Configure the Prometheus data source in Grafana
• Create Grafana-managed alert rules