Project layout

After this page, you will be able to navigate a scaffolded bolt project quickly and find the right folder for each task. You will also see how that project maps to the framework package itself.

The trees below come from a real scaffold in sample_project/ and the installed bolt-pipeliner==0.2.6 package.

Scaffolded project tree

sample_project/
|-- _boltpipeliner/
|   |-- bolt_pipeliner/
|   `-- README.md
|-- configs/
|   |-- etl_config.yaml
|   `-- style_config.yaml
|-- data/
|   `-- flatfiles/
|       `-- example.csv
|-- etl/
|   |-- _flatfile/
|   |   |-- __init__.py
|   |   `-- flatfile_example.py
|   |-- 0_bronze/
|   |   |-- __init__.py
|   |   `-- bronze_example.py
|   |-- 1_silver/
|   |   |-- __init__.py
|   |   `-- silver_example.py
|   `-- 2_gold/
|       |-- __init__.py
|       `-- gold_example.py
|-- macros/
|   `-- __init__.py
|-- tests/
|   `-- test_smoke.py
|-- bolt.py
|-- generate.py
|-- main.py
`-- README.md

Important directories in a scaffolded project:

• configs/: project configuration files. etl_config.yaml controls layers, jobs, storage, and tests.
• data/flatfiles/: local input files for flatfile jobs.
• etl/: transformation code grouped by layer in execution order.
• macros/: reusable Python helpers that jobs can import.
• tests/: project level tests and smoke checks.
• _boltpipeliner/: vendored framework copy when scaffolded in self contained mode.

Important entrypoint files:

• main.py: runs the pipeline from project config.
• generate.py: runs code generators such as docs and notebooks.
• bolt.py: shim for the local vendored CLI.

Framework package tree

.venv/lib/python3.11/site-packages/bolt_pipeliner/
|-- bases/
|-- cli/
|-- config/
|-- generators/
|-- sessions/
|-- templates/
|-- testing/
|-- __init__.py
|-- runner.py
`-- selection.py

Important framework directories:

• bases/: built in base classes for Spark, Pandas, and Polars jobs.
• cli/: command handlers for bolt init, bolt run, bolt test, and bolt generate.
• config/: YAML loading and validation utilities.
• generators/: airflow, docs, layers, and notebook generation logic.
• sessions/: Spark session profile loading and session creation helpers.
• templates/: file templates used by scaffolding and documentation generation.
• testing/: data quality check implementations and test orchestration.

Important framework modules:

• runner.py: job loading and execution flow.
• selection.py: dbt style selection parsing and filtering.