Engineering notes
Design rationale and war stories from building SQE — replacing a Trino fork with Rust, surviving the Iceberg rebase, the caching layers, the DuckDB-shaped surprise, and shipping column-level lineage. Synced from the source repository.
-
A dashboard for the engine
SQE already tracked every query and worker in memory, but there was no way to look at it without tailing logs or wiring Grafana. So we put a read-only dashboard on the health port: no login, no build step, no external assets. One HTML page over a small JSON API, with a tiny in-memory sampler so the charts move over time. Here is how it is built and why it stays deliberately small.
-
Speaking Quack: SQE as a DuckDB server, a DuckDB client, and a federation engine
DuckDB 1.5 ships a wire protocol called Quack. We re-implemented it in pure Rust, turned SQE into both server and client, and proved you can JOIN an Iceberg table with a remote DuckDB table in a single SELECT. Two waves of bugs, one federated query.
-
The type matrix as a roadmap: seven DuckDB types in two days
We started with a markdown table tracking which DuckDB types we could round-trip. The table became the roadmap. DECIMAL, LIST, STRUCT, MAP, ARRAY, ENUM, UNION each got their own MR. The surprises came from how DuckDB models the relationships: MAP is LIST<STRUCT>, UNION is STRUCT with a tag field, and DECIMAL packs four widths into one logical type.
-
Porting DuckDB's BinarySerializer to pure Rust
Ten sub-MRs in a day. The wire format, the fixture-driven debugging loop, and two bugs that the C++ reference encoder ships without telling you: WriteListWithDefault elision, and uninitialised bytes at NULL VARCHAR positions.
-
The SSB regression that wasn't
MR #220 wired runtime filters into iceberg-rust's scan path and dropped TPC-DS 67%. SSB looked like it regressed 6%. Two failed heuristic attempts, one parquet-trace session, and ten warm passes later, the regression turned out to be measurement noise. The fix-the-fix that wasn't, and the data-clustering insight that explains why two suites with the same code path behave nothing alike.
-
q72, our nemesis, and the Int32 that hid for a month
TPC-DS q72 sat at 10 seconds while every other query ran in under 1.4. Five days of investigation chased scan parallelism, range-based NDV, iceberg-rust upgrades, and the RisingWave fork. None of those were the bug. The bug was a silently-skipped Err arm in our dynamic-filter evaluator that swallowed every Int32 vs Int64 type clash. Fixing it: 15.5s to 0.77s. q72 now beats Trino.
-
Nineteen MRs, four waves, and the failure modes of agent batches at scale
Two days after the nine-PR audit pass, we ran a bigger one. 130 issues filed, 19 themed MRs merged across four waves. Same workflow, more failure modes. Watchdog stalls, a reboot mid-wave, a broken main, and config.rs as the conflict magnet. Here is what actually happened.
-
Nine PRs, two merge conflicts, and the value of themed branches
Sunday afternoon: an audit dropped eighteen issues into the tracker. Monday morning we had nine merge requests open. Two of them hit conflicts on rebase. Neither one mattered. Here is why that was deliberate, not lucky.
-
read_parquet shouldn't read /etc/shadow
Modern object-store abstractions unify the filesystem and HTTP behind a single URL. That's a feature for ergonomics. It's a security trap when the URL comes from a user. SQE shipped that trap and then closed it. The IMDS pivot is the part worth telling.
-
Mounting catalogs from SQL: ATTACH, DETACH, and the registry pattern
SQE now ships DuckDB-style ATTACH / DETACH and CREATE / DROP SECRET. The story of building it covers parser extension, credential hygiene, a lifecycle bug we found in the integration tests, and a state-store pattern that is starting to repeat.
-
Shipping OpenLineage: column-level lineage for an Iceberg engine
SQE now emits OL 2-0-2 events with column-level lineage on every write. Here is what they look like, why we walked the LogicalPlan to build them, and the disk spool we did not want to write.
-
How we accidentally created a DuckDB
SQE started as a distributed Iceberg query engine. Five MRs later it queries CSVs from disk, Parquet from S3, and Parquet from HuggingFace. We did not plan that.
-
One Binary, No Cluster: SQE Goes Embedded
We built SQE for distributed Iceberg, but most of the time you just want to look at a parquet file. Here's how we made the engine work both ways without forking the codebase.
-
SQE Talks to Five Catalogs Now: HMS, Nessie, Glue, JDBC, S3 Tables
We claimed the engine was catalog-agnostic. Time to prove it. One branch, five live integration tests, one small AWS SigV4 patch, and a matrix score that moved from 153 to 158.
-
Why a Public Iceberg Matrix Beats Vendor Spec Sheets
Sixty-three capabilities, three levels, no marketing. The Iceberg Matrix is what compatibility looks like when the rubric is public and the evidence has to land in code. Here is why it works, what we learned from sitting on it, and why every open standard needs one.
-
The Iceberg Matrix and the Quiet Bug Hiding in V3
We thought the V3 path worked. The unit tests said it worked. The matrix called it 'partial' and we agreed. Then we wrote eleven end-to-end tests and discovered Polaris had been silently rejecting every V3 column type for months.
-
Our Nemesis: TPC-DS Query 72 and the Limits of a Custom SQL Engine
One query. Ten tables. Twelve times slower than Trino. Everything we tried, what worked, what didn't, and where the ceiling is.
-
DataFusion 53, a Vendored Fork, and 40% Faster Queries
We upgraded SQE from DataFusion 52 to 53 by forking and rebasing iceberg-rust ourselves. The result: 27-40% faster across every benchmark suite.
-
43 Findings, Zero Deferred: A Production Security Audit of a Rust SQL Engine
We ran a full production sign-off audit against SQE and found 43 issues across security, runtime safety, logic bugs, and code quality. Then we fixed all of them in one session.
-
How Agentic AI Helped Us Beat Trino
221 queries, 7 suites, one week — how an AI assistant running automated benchmarks drove a major performance breakthrough.
-
Five Layers of Caching and an 8.8x Speedup Over Trino
How multi-layer caching took SQE from slower than Trino to 2.5-8.8x faster across every benchmark suite.
-
Streaming Writes, Sort Order Safety, and the IN (Subquery) Workaround
Fixing OOM in CTAS, safe Iceberg sort order for mixed writers, and working around DataFusion limitations.
-
From 63% to 95%: Building Trino SQL Compatibility in a Single Day
Implementing 70+ UDFs, Iceberg time travel, metadata TVFs, and engine-level SQL features for Trino drop-in replacement.
-
Building a Comprehensive SQL Benchmark Suite
Seven benchmark suites, 222 queries, and the infrastructure to measure performance honestly.
-
We Replaced Our Trino Fork with a Rust SQL Engine
How we went from maintaining a 2M-line Java fork to shipping a 50MB binary that runs every query as the authenticated user.
-
How We Build Software with AI Assistants
From brainstorm to production in four phases — structured AI collaboration that produces better software than either alone.
-
Making SQE Work Everywhere: Pluggable Auth and Catalogs
How we're turning a single-vendor query engine into something that runs against any identity provider, any catalog, and any cloud.
-
When Your SQL Engine Understands Meaning
SQL engines know table shapes. We're adding ontologies, property graphs, vector search, and AI-native interfaces.