Choosing a runtime
At a glance#
| Standard | DuckDB | Spark | Lakehouse | |
|---|---|---|---|---|
| Engine | Managed query engine, SQL plus pandas Prep | DuckDB 1.5.5 on one machine | Apache Spark 3.5.6 cluster | Spark SQL 4.0 with Iceberg 1.10 |
| Compute size | None | You choose | You choose, plus minimum workers | You choose, plus minimum workers |
| Start | Nothing to start | Quick start, then kept warm for datasets | Cluster start, then kept warm for datasets | Start a session, then run many queries |
| Connections | One SQL source or the lakehouse | Many, in one query | Many, in one job | The warehouse you open |
| Writes | No (read queries) | Yes, with the helper library | Yes, with the helper library | Yes, with SQL on Iceberg tables |
| Scale | Pages of rows | Fits in one machine's memory, or streams from storage | Distributed across workers | Distributed across workers, results paged |
| Languages | SQL and Python (pandas) | Python and SQL | Python (PySpark) and SQL | SQL (Spark dialect) |
| Helper library | None | GZ.DuckDBUtils |
GZ.SparkUtils |
None |
| ML variants | None | scikit-learn, PyTorch, TensorFlow | scikit-learn, PyTorch, TensorFlow | None |
| Best for | Charts, dashboards, quick lookups | Fast analytics, ETL, joins across sources | Large data, wide shuffles, distributed ETL | Exploring and maintaining Iceberg tables with SQL |
Decision guide#
Work down this list and stop at the first match.
- Is it one query on one SQL source or the lakehouse, feeding a chart, dashboard or API read? Use Standard.
- Is the work interactive SQL on Iceberg tables: exploring, creating or
altering tables,
MERGE, time travel, table maintenance, policies? Use the Lakehouse: open a warehouse, start a session and write SQL. - Does it join several connections, write results, run on a schedule, or run Python beyond a few lines of pandas? Continue.
- Does the data fit comfortably in the memory of one machine, or can DuckDB stream it from object storage (Parquet, Iceberg)? Use DuckDB. It starts quickly and needs no cluster.
- Is the data too large for one machine, or does the job need distributed shuffles, joins of very large tables or Spark-native libraries? Use Spark.
- Do you need scikit-learn, PyTorch or TensorFlow? Pick the matching ML variant of DuckDB or Spark.
- Do you need packages that are not installed? Ask an administrator for a custom runtime built on the runtime you picked.
Start small
Prototype on Standard or DuckDB. Move to Spark when a job outgrows one machine, not before. The Groundzero helper libraries of DuckDB and Spark follow the same shape, which keeps a port short.
Where the choice is made#
You choose a runtime in three places: when you create a dataset, when you create or schedule an ETL job, and when you start a notebook. The runtime control lists the runtimes available to you and, except for Standard, a compute size. Standard appears only in the dataset designer. You choose the Lakehouse by opening a warehouse in Lakehouse, where you pick a compute size and start a session.