Examples walkthrough
After this page, you will know what each official example project demonstrates and which one to start with for your setup. You will also have verified commands for running the lightweight local examples first.
The examples live in the framework source repository under examples/:
git clone https://github.com/ormenesse/boltdbt_pipeliner.git
cd boltdbt_pipeliner/examplesWhich example should you pick
| Project | Engine and storage | Layers | Best use |
|---|---|---|---|
demo/ | PySpark + Iceberg | flatfile, bronze, silver, gold | Small Spark and Iceberg medallion walkthrough with a single silver fact table. |
demo_spark/ | PySpark + local Parquet | flatfile, bronze, silver, gold, diamond | End to end Spark local run with NYC collisions data and a Spark ML model. |
demo_pandas/ | Pandas + local Parquet | flatfile, bronze, silver, gold, diamond | Local single machine run of the NYC pipeline with a NumPy logistic model. |
demo_polars/ | Polars + local Parquet | flatfile, bronze, silver, gold, diamond | Local single machine run of the same NYC pipeline with Polars transforms. |
demo/: minimal Spark and Iceberg reference
Use this when you want the smallest medallion style project that still shows incremental policy and tests.
cd examples/demo
bolt run --silver --config configs/etl_config.yaml
bolt test --config configs/etl_config.yaml
bolt generate documentation --config configs/etl_config.yamlThis example expects a working local Spark and Iceberg setup.
demo_spark/: full Spark local pipeline
Use this when you want Spark feature engineering and model training on a realistic public dataset.
cd examples/demo_spark
python scripts/download_nyc_collision_data.py --limit 2500
bolt run --config configs/etl_config.yaml --verbosedemo_pandas/ and demo_polars/: lightweight local paths
Use these when you want fast local iteration without a Spark cluster.
cd examples/demo_pandas
python scripts/download_nyc_collision_data.py --limit 50
bolt run --config configs/etl_config.yaml --flatfile --bronzecd examples/demo_polars
python scripts/download_nyc_collision_data.py --limit 50
bolt run --config configs/etl_config.yaml --flatfile --bronzeThe commands above were verified in this docs workspace with bolt-pipeliner==0.2.6 and local fixture data.
For full medallion and diamond runs in these two demos, verify with the latest framework source checkout. In this 0.2.6 environment, the current repository demo configs can raise a year_month column error during the later silver step.
Practical starting point
Start with demo_pandas/ or demo_polars/ to validate your local environment quickly. Move to demo_spark/ after that, then use demo/ when you want to see the Spark and Iceberg medallion pattern in a compact project.