ETL jobs

An ETL job is Python code that reads data from your connections, transforms it and writes the result somewhere else. After this chapter you can create a job, run it, read its log, stop it, and share it with other projects.

Jobs live in a project, on the Jobs tab. A job can also run on a schedule (see Scheduling and run history).

Is a job what you need?#

You need a project, at least one connection, permission to run ETL jobs and a compute size you may use. Jobs never run on Standard.

Create a job#

1. In a project, open the Jobs tab and select New Job.
2. Type a Job Title that follows the name rule.
3. Select Add.

What happens next: You see Job Added successfully, and the job appears in the list. Select its name to open the editor.

The list shows each job's name, who saved it last and when. The row menu has Rename (Job renamed successfully), Delete (Job deleted successfully) and Export.

Write and run a job#

By the end of this task your job has run and you have read its log.

1. Select the job name to open the editor.
2. In the left drawer, open Connections and tick the connections the job uses. Your code can use only the connections you tick.
3. Write your code (see the examples below). The editor suggests connection, database, table and column names as you type.
4. Open the runtime picker and choose a Runtime and Compute size. See Runtimes and compute.
5. Select Run. Groundzero saves the job first. If the save fails, nothing runs.
6. Watch the Messages pane. It fills with log lines while the job runs.

How to check it worked: The message ETL job submitted successfully appears, and the log ends with Job completed successfully. If it ends with Job failed. Check the logs for the cause., read the last log lines.

The editor toolbar#

The left drawer#

Write job code#

Every job starts by creating a session, then reads, transforms and writes through the connection names you ticked.

A Spark job#

from GZ.SparkUtils import sparkSession

spark = sparkSession("DailySalesRollup", "WARN")

orders = spark.readJDBC(
    connName="AcmePOSPostgres",
    query="SELECT store_id, order_date, total_amount FROM public.orders",
)

daily = (
    orders.groupBy("store_id", "order_date")
    .sum("total_amount")
    .withColumnRenamed("sum(total_amount)", "revenue")
)

spark.writeGzLakehouse(
    df=daily,
    connName="AcmeLakehouse",
    tableName="sales.daily_sales",
    mode="overwrite",
)

sparkSession(appName, logLevel="WARN") returns a Spark session with extra methods that read from and write to your connections. Give it a meaningful application name.

A DuckDB job#

from GZ.DuckDBUtils import duckSession

session = duckSession()
sales = session.readGzLakehouse(
    connName="AcmeLakehouse", tableName="sales.daily_sales"
)
session.register("sales", sales)
top = session.query("SELECT store_id, SUM(revenue) AS r FROM sales GROUP BY 1")
result_df = top.df()

Read and write other systems#

Each connection type has its own read and write methods. You pass the connection name as it appears on the Connections page, never a password.

Common DuckDB session methods are readGzLakehouse(connName, tableName), readS3File(connName, bucketName, filePath, fileType), register(name, relation) and query(sql). Call .df() on a result to get a pandas DataFrame.

Not sure of the exact syntax? Open Utilities in the left drawer. It has templates for moving data between S3, GCS, Azure Blob, BigQuery, Snowflake, Redshift and GZ Lakehouse, such as S3 to GZLakehouse and Snowflake to S3. Replace each {{PLACEHOLDER}} value and any optional parameters in the comments. You can also ask the code assistant, or your MCP client, to look up the syntax for a connection type.

Use system variables#

System variables are placeholders in the form ${NAME}. Groundzero fills them in just before your code runs.

What your code may not do#

Job code runs in a protected space on the Spark and DuckDB runtimes. This keeps credentials safe.

Run jobs together#

Run other jobs along with this one#

1. Open Jobs in the left drawer.
2. Pick other jobs from the same project.
3. Select Run.

Groundzero runs the picked sub-jobs together with the main job and gives each its own saved connections. A run can include up to 255 sub-jobs, and each job can appear once. This needs a Spark, DuckDB or pandas runtime. Otherwise the run fails with Invalid RTE for orchestration.

Load models#

Open Models in the left drawer and pick registered models. They are loaded before your code starts. See Models and MLflow.

Read the log and stop a run#

The Messages pane shows lines about once a second while the job is logging. The download icon above the pane saves the log lines.

To stop a run, select the red Stop icon. Job aborted successfully confirms it. If a run cannot be stopped, a message gives the reason.

If the run does not start#

Share a job#

Published jobs form a shared library for your site.

Publish a job#

1. In the Jobs list, open the row menu and select Export.
2. If no published job has that name, you see name exported successfully.
3. If one exists, the Override Export dialog asks whether to replace it. Choose Override to replace it, or cancel.

Import a job#

1. On the Jobs tab, select Import Job.
2. Choose a published job in Select a job and select Import.
3. If your project already has a job with that name, the Job Name Conflict dialog offers Replace, Keep Both (adds a timestamp to the new name) and Cancel.

Administrators can review and delete published jobs, and import them across sites, in the admin console.

Troubleshooting#