Overview

What the DuckDB runtime gives you#

A Groundzero runtime is a software environment plus a compute size. When you run a dataset, job or notebook, Groundzero starts the runtime you selected with the connections your code needs, runs your code, and returns the output or logs.

The DuckDB runtime includes:

• Python 3.12 with pandas, NumPy and PyArrow.
• DuckDB 1.5.5 with the iceberg, httpfs, aws, postgres, mysql, azure, json, parquet, spatial and motherduck extensions already installed, so nothing is downloaded when your code runs.
• The Groundzero helper library (GZ). duckSession() returns a session object that wraps a DuckDB connection and adds one read and write method per connector (S3, JDBC databases, Snowflake, BigQuery, the Groundzero Lakehouse, REST APIs, vector stores, LLM providers and more).
• PDF text extraction (Docling with RapidOCR), the Qualipilot data quality library, and clients for Groundzero Agents and MCP servers.
• JupyterLab for notebooks, with numpy, pandas, boto3, getClient and duckSession already imported.

DuckDB is a columnar engine that runs inside your Python process. There is no cluster to size and no copying between Python and the engine. Relations, pandas DataFrames and Arrow tables are queried in place, and work larger than memory spills to local disk.

Choose the DuckDB runtime#

Groundzero offers three runtime choices. Pick the one that matches the shape of your workload.

Choose DuckDB when:

• the data you actually process fits one machine's memory and local disk;
• you join several sources (for example a lakehouse table, an S3 file and a PostgreSQL table) in one SQL statement;
• you want a quick start for datasets and previews, because DuckDB needs no cluster;
• you want to mix SQL with pandas, scikit-learn or Arrow code.

Choose another runtime when:

• you must write to the Groundzero Lakehouse. DuckDB reads the lakehouse only, so use the Spark runtime for lakehouse writes;
• the data does not fit on one machine even after filtering and projection;
• the dataset is a single SQL query against one SQL connection and needs no code, which the Standard option covers.

How a run works#

Every dataset, job or notebook that uses the DuckDB runtime runs in its own isolated environment. Its size comes from the compute you choose in the runtime selector (see the Quickstart).

A session handles one run at a time. For datasets and previews, Groundzero keeps idle sessions ready for a short while. A new run uses an idle session if one exists and otherwise starts another, so runs never queue behind each other. An idle session is released after about five minutes. Do not keep state in memory between runs, because any run may land on a new session.

Credentials#

Your code only names a connection (connName="sales_s3"). It never handles keys or passwords. Secret values are hidden from your code: os.environ shows them as ****.

Resources#

The compute you pick sets the CPU and memory of the run. DuckDB is sized from it automatically:

• threads equal the CPU count of the compute (minimum two);
• the DuckDB memory limit is 80 percent of the compute's memory (minimum 512 MB), which leaves the rest for Python, Arrow and client libraries;
• work that does not fit in memory spills to disk under /tmp/duckdb-spill.

To override threads and memory for one session, call duckSession(threads=..., memory_limit=...).

Versions#

The runtime is updated continuously and has no numbered releases.

Packages without a version in these tables install at their latest release when the runtime is built. The complete list is in Installed packages.

What changed recently#

Recent changes that affect your code, newest first:

• Lakehouse reads in long-running sessions no longer fail with expired storage tokens.
• Lakehouse access policies, including column masking, apply to the user who runs the code. Cached rows are kept per user policy.
• Lineage records schemas for connector reads and writes.
• DuckDB threads and memory are sized from your compute, and each dataset session uses the full compute for one run at a time.
• Time zone data is included, so time-zone-aware timestamps (including UTC) work.
• Python 3.12 is the base for every runtime, including the ML variants.

Machine-learning variants#

Three variants add machine learning libraries on top of the DuckDB runtime. They share the same GZ library, extensions and behavior.

scikit-learn, NumPy, SciPy and pandas are in every variant. The PyTorch variant uses the CUDA 12.1 build of PyTorch, so it can use an NVIDIA GPU when you set a GPU size.

If a library is in none of the variants, ask an administrator to create a custom runtime (see Use extra packages).