Observability: metrics + Grafana

Goal

SQE exposes Prometheus metrics on its metrics.prometheus_port (/metrics, port 9090 in this quickstart). This stack scrapes them with VictoriaMetrics and renders them in Grafana, giving you a live view of query rate, cache hit/miss, active sessions, scan pruning, and coordinator memory while queries run.

A minimal queryable SQE (Nessie catalog + RustFS, anonymous auth) sits underneath just to generate real metrics — the focus here is the monitoring pipeline, not the data layer.

Components

Configuration

Backend (sqe.toml)

[coordinator]
flight_sql_port = 50051
trino_http_port = 8080
mode = "hybrid"

[worker]
memory_limit = "4GB"

[[auth.providers]]
type = "anonymous"
user = "anonymous"
roles = ["admin"]

[catalogs.nessie]
polaris_url = "http://nessie:19120/iceberg"
warehouse = "warehouse"

[storage]
s3_endpoint = "http://rustfs:9000"
s3_region = "eu-example-1"
s3_access_key = "s3admin"
s3_secret_key = "s3adminpw"
s3_path_style = true
s3_allow_http = true

[metrics]
prometheus_port = 9090

SQL (queries.sql)

CREATE SCHEMA IF NOT EXISTS nessie.demo;

DROP TABLE IF EXISTS nessie.demo.events;
CREATE TABLE nessie.demo.events (
    id     BIGINT,
    kind   VARCHAR,
    amount DOUBLE
);

INSERT INTO nessie.demo.events VALUES
    (1, 'click',    1.50),
    (2, 'purchase', 42.00),
    (3, 'click',    0.75),
    (4, 'purchase', 13.25);

SELECT kind, COUNT(*) AS n, ROUND(SUM(amount), 2) AS total
FROM nessie.demo.events
GROUP BY kind
ORDER BY total DESC;

The test

run.sh brings the full stack up (docker compose up -d --wait), then runs queries.sql three times via sqe-cli to populate real metric counters. It polls the VictoriaMetrics API until sqe_rows_returned_total appears (up to ~45 s, 5 s scrape interval), then queries each key metric via the /api/v1/query endpoint and asserts the scrape target is up{job="sqe-coordinator"} = 1. Finally it curls sqe:9090/metrics directly and captures a sample of raw sqe_* counters to OUTPUT.md. Open Grafana at http://localhost:13000 (admin / admin) for the provisioned SQE Overview dashboard. Tear down with ./run.sh --down.

Output

# Captured output

_Generated by `./run.sh` on 2026-06-07T09:12:02Z._

## VictoriaMetrics is scraping the SQE coordinator
scrape target up{job="sqe-coordinator"} = 1
sqe_rows_returned_total                    = 9
sqe_active_sessions                        = 0
sqe_cache_misses_total                     = 0
sqe_coordinator_memory_used_bytes          = 0

## A sample of SQE's raw /metrics (sqe_*)
sqe_active_sessions 0
sqe_cache_hits_total 0
sqe_cache_misses_total 0
sqe_coordinator_memory_used_bytes 0
sqe_files_pruned_minmax_total 0
sqe_rows_returned_total 27
sqe_s3_bytes_read_total 0