Sovereign Query Engine
Roadmap

Where SQE is, and what’s next.

SQE is in active development. This is the current state, the milestones that got it here, and the verified open roadmap — the full commit history lives on GitHub.

88.4% Iceberg matrix (167/189) — top five
6/7 benchmark suites won vs Trino 465
222/222 queries pass at SF1
43/43 security audit findings resolved
Milestones

Recently shipped

  1. Jun 2026

    Read-only ops web UI

    A network-gated dashboard ships inside the coordinator binary — live queries with per-fragment timing, cluster workers, and engine metrics with 12h history (sparklines, gauges, a query-activity histogram). No login, no build step, no external assets.

  2. May 2026

    One distributed engine

    After driving an Apache Ballista integration to functional parity and measuring it honestly — slower where it completed, couldn’t finish the TPC-DS analytical core — we removed it. The bespoke scheduler (weighted placement, scan locality, straggler handling) is the only distributed engine.

  3. May 2026

    Security & correctness audit

    A multi-wave audit campaign hardened the stack: a sealed secret type, per-user session isolation and cache keying, policy correctness on DELETE/UPDATE, write-path cleanup on cancel, and constant-time credential checks. Every audit finding resolved.

  4. May 2026

    DuckDB-class embedded mode

    File-format table functions (read_parquet / csv / json / avro), SELECT * FROM 'file' auto-detect, COPY … TO, a Delta Lake reader, and HuggingFace hf:// URLs. The embedded persona that turned out DuckDB-shaped.

  5. May 2026

    Column-level lineage

    A native OpenLineage 2-0-2 emitter with column-level lineage on writes. File and HTTP sinks, disk-spool fallback for collector outages. Off by default.

  6. Apr 2026

    Five catalogs, live-verified

    Apache Polaris, Project Nessie, Unity Catalog OSS, AWS Glue, and AWS S3 Tables — each behind live integration tests. AWS SigV4 support added to the vendored REST catalog for federated endpoints.

  7. Apr 2026

    Runtime filter pushdown

    DataFusion 53 runtime filters from hash-join build sides now feed the Iceberg scan’s row-group, page-index, and row-filter pruning — double-digit TPC-H speedups at SF1 and SF10.

  8. Apr 2026

    DataFusion 53.1 + vendored Iceberg fork

    ~40× faster planning, hash-join dynamic filters, and three filter-pushdown fixes. A vendored iceberg-rust fork rebased on DataFusion 53 with a SigV4 feature for federated REST.

  9. Q1–Q2 2026

    Iceberg matrix: 52% → 88.4%

    Phased coverage work lifted the public Iceberg matrix score from a 52% baseline to 88.4% (167/189) — V3 features end to end, partition evolution, merge-on-read deletes, bloom filters, and live catalog verification.

Design rationale for many of these lives in the book, Sovereign by Design, and in the engineering blog in the source repository.

Roadmap

What’s open

Verified against the current state — features that have shipped are not listed here. Grouped by where each item stands today.

In progress4

  • HuggingFace glob (hf://…/**/*.parquet) A custom object store so globs enumerate files via the HF tree API and the SQL pre-rewriter retires.
  • Copy-on-write DML at TPC-E SF100 Parallelise per-file rewrite + streaming writes to keep the heaviest update under the harness cap.
  • Snowflake Horizon catalog — live test REST-compatible today; a live integration test against a real Horizon endpoint is the open item.
  • Helm chart for Kubernetes A first-class chart for the coordinator + worker topology.

Planned5

  • Local data-file block cache Alluxio-style caching of hot data files on workers.
  • Iceberg Puffin bloom-filter reading Use Puffin sidecar bloom filters for additional scan pruning.
  • Sort-on-write enforcement A writer pass that physically sorts files to the table’s sort order, not just the metadata.
  • Smart-CSV byte sampling Sample bytes to infer delimiter/quote/header, beyond today’s extension heuristics.
  • Semantic AI layer RDF/SPARQL, property-graph, and vector-search surfaces over the same engine.

Blocked upstream5

  • Iceberg V3 Variant & shredded Variant Blocked upstream in iceberg-rust / arrow-rs.
  • Iceberg V3 Geometry types Needs a DataFusion user-defined type.
  • Iceberg V3 Vector / Embedding types Spec not finalised.
  • Iceberg V3 row lineage Deferred upstream.
  • Multi-argument partition transforms (V3) Spec not stable.