Loki: ship and search logs

Loki is a log store built by the Grafana team that indexes only labels (host, unit, job), not the text of every line. That makes it cheap to run and simple to query: pick a label set, then filter and parse the lines. An agent ships the logs; the current one is Grafana Alloy, which replaced Promtail. This guide gets all three pieces running on one host and shipping that host’s own logs.

Step 1: the compose file

Create a directory, save this as compose.yaml:

services:
  loki:
    image: grafana/loki:latest
    command: -config.file=/etc/loki/local-config.yaml
    ports:
      - "3100:3100"
    volumes:
      - loki-data:/loki

  alloy:
    image: grafana/alloy:latest
    command: run --server.http.listen-addr=0.0.0.0:12345 --storage.path=/var/lib/alloy/data /etc/alloy/config.alloy
    ports:
      - "12345:12345"
    volumes:
      - ./config.alloy:/etc/alloy/config.alloy:ro
      - /var/log:/var/log:ro
      - /run/log/journal:/run/log/journal:ro
      - /etc/machine-id:/etc/machine-id:ro
      - alloy-data:/var/lib/alloy/data
    depends_on:
      - loki

  grafana:
    image: grafana/grafana-oss:latest
    ports:
      - "3000:3000"
    volumes:
      - grafana-data:/var/lib/grafana
    depends_on:
      - loki

volumes:
  loki-data:
  alloy-data:
  grafana-data:

Loki uses its built-in single-binary configuration from the image. Alloy gets read-only access to /var/log and the journal; /etc/machine-id is needed so the journal reader can find the right journal directory.

Step 2: the Alloy configuration

Save this as config.alloy in the same directory:

local.file_match "varlog" {
  path_targets = [{
    __path__ = "/var/log/*.log",
    job      = "varlog",
    host     = constants.hostname,
  }]
}

loki.source.file "varlog" {
  targets    = local.file_match.varlog.targets
  forward_to = [loki.write.default.receiver]
}

loki.relabel "journal" {
  forward_to = []

  rule {
    source_labels = ["__journal__systemd_unit"]
    target_label  = "unit"
  }
}

loki.source.journal "journal" {
  max_age       = "12h"
  relabel_rules = loki.relabel.journal.rules
  labels        = {job = "systemd-journal", host = constants.hostname}
  forward_to    = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

• local.file_match expands the glob and attaches static labels; loki.source.file tails those files and remembers positions across restarts in the storage path.
• loki.source.journal reads the systemd journal. Journal fields arrive as internal labels starting with __journal_; the loki.relabel block copies the unit name into a real unit label so you can query by service.
• loki.write is the destination. Every forward_to points at its receiver, which is how Alloy pipelines are wired together.
• Keep the label set small: host, job, unit. Labels with thousands of values (request IDs, user IDs) belong in the line, not in labels, or Loki slows down.

Step 3: start it and check

docker compose up -d
docker compose logs alloy --tail 20
curl -s http://localhost:3100/ready
curl -s -G http://localhost:3100/loki/api/v1/labels | python3 -m json.tool

/ready answers ready once Loki accepts writes (it may take a few seconds after start). The labels call should list host, job and unit. Alloy’s own UI at http://localhost:12345 shows every component, its health and the targets it discovered, which is the first place to look if nothing arrives.

Step 4: search in Grafana

Open http://localhost:3000, add a Loki data source with URL http://loki:3100, then go to Explore, choose Loki, switch the editor to Code and try these LogQL queries:

{job="systemd-journal", unit="sshd.service"}
{job="systemd-journal"} |= "Failed password"
{job="varlog"} |~ "(?i)error|fail" != "health"
{job="systemd-journal"} | json | level="error"
sum by (unit) (count_over_time({job="systemd-journal"} |= "error" [5m]))
rate({job="varlog"} |= "error" [1m])

• The part in braces is the stream selector and must contain at least one label; it decides which chunks Loki reads.
• |= and != filter lines by substring, |~ and !~ by regular expression.
• | json and | logfmt parse structured lines into fields you can filter on with field="value".
• count_over_time and rate turn log lines into metrics; switch Explore to a graph to see error counts per unit over time, and use the same expressions in dashboards and alert rules.

Step 5: the same from the terminal

logcli ships with Loki’s releases and is handy on servers without a browser:

export LOKI_ADDR=http://localhost:3100
logcli labels job
logcli query '{job="systemd-journal", unit="sshd.service"}' --since=1h --limit=50
logcli query '{job="varlog"} |= "error"' --tail

Adding more hosts

Run only the alloy service on every other machine with the same config.alloy, changing the loki.write URL to the central Loki host. Because the host label comes from constants.hostname, streams from different machines stay separate automatically. Put Loki behind a reverse proxy with TLS and authentication before exposing it beyond a trusted network; Loki itself has no built-in authentication.

Official documentation

• Loki: Install with Docker
• Alloy: loki.source.journal
• LogQL: Log query language