Sovereign Query Engine
Roadmap

Where SQE is, and what’s next.

SQE is in active development. This is the current state, the milestones that got it here, and the verified open roadmap — the full commit history lives on GitHub.

90.7% Iceberg matrix (147/162), current rubric
8/8 benchmark suites faster than Trino 481 at SF1 (Oct 2026)
6/7 suites faster at SF10 (Jul 2026)
43/43 security audit findings resolved
Milestones

Recently shipped

  1. Oct 2026

    Iceberg V3 by default

    New tables are format-version 3. A JSON column is stored as an Iceberg VARIANT and reads back as JSON text; shredded variant files written by other engines (Spark) are read on the coordinator and on workers. Merge-on-read DELETE writes deletion vectors, and compaction triggers on the deleted-row share. Row lineage is maintained at write time, and V3 unknown columns are supported. Rescored against the current Iceberg matrix rubric: 147/162 (90.7%).

  2. Aug 2026

    MCP, governed by the caller's own token

    SQE speaks Model Context Protocol over Streamable HTTP from the same coordinator binary, off by default. It is an OAuth resource server rather than a credential proxy: the caller's bearer becomes the same per-user session Flight SQL uses and travels to the catalog and storage unchanged, so an assistant reads exactly what its human can read. The read-only SQL tool is gated by SQE's own parser as an allow-list, and the write tool needs an OAuth scope, an engine role, and source authorization independently.

  3. Jun 2026

    Read-only ops web UI

    A network-gated dashboard ships inside the coordinator binary, live queries with per-fragment timing, cluster workers, and engine metrics with 12h history (sparklines, gauges, a query-activity histogram). No login, no build step, no external assets.

  4. May 2026

    One distributed engine

    After driving an Apache Ballista integration to functional parity and measuring it honestly, slower where it completed, couldn't finish the TPC-DS analytical core, we removed it. The bespoke scheduler (weighted placement, scan locality, straggler handling) is the only distributed engine.

  5. May 2026

    Security & correctness audit

    A multi-wave audit campaign hardened the stack: a sealed secret type, per-user session isolation and cache keying, policy correctness on DELETE/UPDATE, write-path cleanup on cancel, and constant-time credential checks. Every audit finding resolved.

  6. May 2026

    DuckDB-class embedded mode

    File-format table functions (read_parquet / csv / json), SELECT * FROM 'file' auto-detect, COPY … TO, and HuggingFace hf:// URLs. The embedded persona that turned out DuckDB-shaped.

  7. May 2026

    Column-level lineage

    A native OpenLineage 2-0-2 emitter with column-level lineage on writes. File and HTTP sinks, disk-spool fallback for collector outages. Off by default.

  8. Apr 2026

    Five catalogs, live-verified

    Apache Polaris, Project Nessie, Unity Catalog OSS, AWS Glue, and AWS S3 Tables, each behind live integration tests. AWS SigV4 support added to the vendored REST catalog for federated endpoints.

  9. Apr 2026

    Runtime filter pushdown

    DataFusion 53 runtime filters from hash-join build sides now feed the Iceberg scan's row-group, page-index, and row-filter pruning, double-digit TPC-H speedups at SF1 and SF10.

  10. Apr 2026

    DataFusion 53.1 + vendored Iceberg fork

    ~40x faster planning, hash-join dynamic filters, and three filter-pushdown fixes. A vendored iceberg-rust fork rebased on DataFusion 53 with a SigV4 feature for federated REST.

  11. Q1-Q2 2026

    Iceberg matrix: 52% -> 88.4%

    Phased coverage work lifted the public Iceberg matrix score from a 52% baseline to 88.4% (167/189, on the rubric of the time): V3 features end to end, partition evolution, merge-on-read deletes, bloom filters, and live catalog verification.

Design rationale for many of these lives in the book, Sovereign by Design, and in the engineering blog in the source repository.

Roadmap

What’s open

Verified against the current state — features that have shipped are not listed here. Grouped by where each item stands today.

In progress2

  • Copy-on-write DML at TPC-E SF100 Parallelise per-file rewrite + streaming writes to keep the heaviest update under the harness cap.
  • Snowflake Horizon catalog, live test REST-compatible today; a live integration test against a real Horizon endpoint is the open item.

Planned5

  • Variant path functions <code>variant_get</code>-style extraction on V3 <code>VARIANT</code> columns; today a variant reads back as JSON text for the <code>json*</code> functions.
  • Local data-file block cache Alluxio-style caching of hot data files on workers.
  • Iceberg Puffin bloom-filter reading Use Puffin sidecar bloom filters for additional scan pruning.
  • Sort-on-write enforcement A writer pass that physically sorts files to the table's sort order, not just the metadata.
  • Smart-CSV byte sampling Sample bytes to infer delimiter/quote/header, beyond today's extension heuristics.

Blocked upstream2

  • Iceberg V3 Geometry types Needs a DataFusion user-defined type.
  • Multi-argument partition transforms (V3) Spec not stable.