Spark runtime
Distributed Apache Spark: the sparkSession, the Groundzero helper API, connectors, packages, limits and a gallery of working examples.
Chapters
- IntroductionThe Groundzero Spark runtime runs Python jobs, dataset queries and notebooks on Apache Spark. A job is a Python script. It gets one ready-made sparkSession, a set of…
- OverviewYou do not install or deploy anything. Pick the Spark runtime and a compute size when you create a job, dataset or notebook.
- QuickstartThis quickstart creates a Spark job that builds a small DataFrame, aggregates it, writes the result to object storage and exposes it as the job output. It uses fictional…
- How-to guidesConnections are created once in Groundzero (credentials stay in the connection) and referenced by name from code. Every connector adds read*, execute* and write* methods…
- API referenceThis section documents the Groundzero helper API that is available in the Spark runtime. Standard PySpark (DataFrame, functions, types, MLlib) is documented by Apache…
- Installed packagesPackages marked "pinned" have a fixed version. The others install at their latest version when the runtime is built. To read the exact version in your session, run pip…
- Limits and troubleshootingJob scripts run with restrictions. Notebook kernels do not apply these rules.
- Examples galleryAll examples use fictional names and data. Replace connection names (acme-*) with your own. Each example is a complete job script.