How to Set Up Prometheus and Grafana for Monitoring

The short version: how it works

Prometheus polls (scrapes) HTTP /metrics endpoints every few seconds and stores the numbers in a time-series database. Exporters expose those metrics: node_exporter covers the server (CPU, memory, disk, network), and your application exposes requests, errors and latency. Grafana connects to Prometheus as a data source and draws the charts.

The minimal working setup is four containers: Prometheus, Grafana, node_exporter and the application itself.

Step 1. Docker Compose

services:
  prometheus:
    image: prom/prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prom-data:/prometheus
    ports:
      - "9090:9090"

  node-exporter:
    image: prom/node-exporter
    pid: host
    volumes:
      - /:/host:ro,rslave
    command: ["--path.rootfs=/host"]

  grafana:
    image: grafana/grafana
    volumes:
      - grafana-data:/var/lib/grafana
    ports:
      - "3000:3000"

volumes:
  prom-data:
  grafana-data:

In production, pin specific image versions and do not expose ports 9090 and 3000 publicly — serve Grafana through a reverse proxy with HTTPS.

Step 2. Prometheus configuration

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["prometheus:9090"]
  - job_name: node
    static_configs:
      - targets: ["node-exporter:9100"]
  - job_name: app
    static_configs:
      - targets: ["app:8080"]

Run docker compose up -d and open http://localhost:9090/targets. Every target should be UP. If one is DOWN, check the service name, the port and that the containers share a network.

Step 3. Application metrics

Most languages have official or widely used Prometheus client libraries. A Node.js example with prom-client:

import express from "express";
import client from "prom-client";

const app = express();
client.collectDefaultMetrics();

const httpDuration = new client.Histogram({
  name: "http_request_duration_seconds",
  help: "HTTP request duration",
  labelNames: ["method", "route", "status"],
});

app.use((req, res, next) => {
  const end = httpDuration.startTimer();
  res.on("finish", () =>
    end({ method: req.method, route: req.route?.path ?? "unknown", status: res.statusCode })
  );
  next();
});

app.get("/metrics", async (_req, res) => {
  res.set("Content-Type", client.register.contentType);
  res.end(await client.register.metrics());
});

app.listen(8080);

The key rule for labels: never put high-cardinality values in labels — user IDs, full URLs, emails. Every unique label combination is a separate time series, and storage grows fast.

Step 4. Basic PromQL queries

Remember: counters (_total, _count) almost always go through rate(), otherwise you get an ever-growing line.

Step 5. Your first Grafana dashboard

1. Open Grafana on port 3000 and change the admin password right away.
2. Connections → Data sources → Prometheus, URL: http://prometheus:9090.
3. Create a dashboard and add panels using the queries from the table.
4. For server metrics, import a ready-made Node Exporter dashboard from the Grafana catalog by its ID.
5. Add an instance variable to switch between servers.

Describe the data source and dashboards through provisioning (YAML and JSON in the repository) so they can be restored and versioned.

Common mistakes

• No alerts. Nobody watches dashboards at night. At minimum, alert on disk usage, unreachable targets (up == 0) and rising errors.
• Retention left to chance. Prometheus keeps data for a limited time by default; set --storage.tsdb.retention.time deliberately.
• Open ports. /metrics and an unauthenticated Prometheus should never be reachable from the internet.
• Too many labels. Cardinality is the main reason Prometheus becomes heavy.