Grafana: your first dashboard from Prometheus data

Prometheus collects the numbers; Grafana turns them into something a human can read at a glance. This guide assumes a running Prometheus that scrapes at least one node_exporter (the one-server stack from the tutorials section is enough) and walks through building a host overview dashboard from scratch, then shows how to keep it as a file.

Step 1: start Grafana and add the data source

docker run -d --name grafana -p 3000:3000 -v grafana-data:/var/lib/grafana grafana/grafana-oss:latest

Open http://localhost:3000, log in with admin / admin and set a new password when asked. Then Connections > Data sources > Add data source > Prometheus. Set the URL to where Grafana can reach Prometheus, for example http://prometheus:9090 inside a Docker network or http://192.0.2.10:9090 across the LAN, and press Save & test. A green message confirms Grafana can query it. If you prefer configuration as code, drop this file into /etc/grafana/provisioning/datasources/ and Grafana creates the data source at start:

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true

Step 2: a new dashboard and the first panel

Dashboards > New > New dashboard > Add visualization, pick Prometheus. In the query editor switch to Code mode and enter the CPU usage query:

100 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100

rate(...[5m]) turns the ever-growing CPU counter into per-second usage over five minutes; averaging the idle share per instance and subtracting from 100 gives busy percent. On the right, set Standard options > Unit to Percent (0-100), set Min 0 and Max 100, and under Thresholds add a red step at 90. Give the panel the title CPU usage and click Back to dashboard.

Step 3: three more panels

Add a panel for each query below. Use the Time series visualization for the first two and Gauge for the disk one.

node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100

rate(node_network_receive_bytes_total{device!~"lo|veth.*|docker.*"}[5m]) * 8

(1 - node_filesystem_avail_bytes{mountpoint="/",fstype!="tmpfs"} / node_filesystem_size_bytes{mountpoint="/",fstype!="tmpfs"}) * 100

• Memory: percentage available; unit Percent (0-100). Available is a better signal than free, because the kernel uses free memory for cache on purpose.
• Network: bytes per second times 8 gives bits per second; unit bits/sec (SI). The regex drops loopback and virtual interfaces.
• Disk: percentage used on the root filesystem; a gauge with thresholds at 80 (orange) and 90 (red) shows the state without reading numbers.
• In the Legend field use {{instance}} or {{device}} so the series are named by label instead of by the raw query.

Drag the panels into a 2 by 2 grid by their titles and resize from the lower right corner. Press Save dashboard (the disk icon), name it Host overview.

Step 4: a variable to switch hosts

With several servers a dropdown beats one dashboard per host. Open the dashboard Settings > Variables > New variable: type Query, name instance, data source Prometheus, query:

label_values(node_uname_info, instance)

Enable Multi-value and Include All option. Then edit each panel and add instance=~"$instance" inside the curly braces of its query, for example node_cpu_seconds_total{mode="idle", instance=~"$instance"}. The dropdown now appears at the top of the dashboard and filters every panel.

Step 5: export it as JSON and provision it

A dashboard that exists only in the browser is lost with the container. Share > Export > Export as JSON, save it as host-overview.json, and tell Grafana to load it from disk with a provider file in /etc/grafana/provisioning/dashboards/:

apiVersion: 1
providers:
  - name: files
    type: file
    options:
      path: /var/lib/grafana/dashboards

Mount the folder with your JSON files at that path and the dashboard is recreated on every fresh start. Keep the JSON in Git next to your Prometheus configuration.

Shortcut: import a community dashboard

For node_exporter the community dashboard Node Exporter Full (ID 1860 on grafana.com) is far more complete than anything built in an afternoon. Dashboards > New > Import, enter 1860, choose your Prometheus data source. Building one panel yourself first is still worth it, because that is how you learn to fix a community dashboard when its queries do not match your exporter version.

Official documentation

• Get started with Grafana and Prometheus
• Grafana: Variables
• Grafana: Provisioning