Code generation

After this page, you will know how to use bolt generate to produce runnable artifacts from your ETL config. You will also understand how the generators order jobs based on dependencies.

bolt generate reads configs/etl_config.yaml by default and writes files under ./outputs.

Command shape

bolt generate TARGETS... [--config PATH]

• TARGETS... is required.
• Valid targets are airflow, documentation, layers, notebook, and all.
• --config, -c lets you point to a different YAML file. The default is configs/etl_config.yaml.

Examples:

# Generate Airflow scripts and docs only.
bolt generate airflow documentation
 
# Generate everything.
bolt generate all
 
# Use a custom config file.
bolt generate notebook --config configs/prod_etl.yaml

Target reference

airflow

Creates Python job scripts and one DAG per layer:

• Job scripts: ./outputs/airflow/code/<layer>_<output_table_name>.py
• DAGs: ./outputs/airflow/dags/datamart_<layer>.py

documentation

Creates static HTML documentation and a schema helper script:

• Main page: ./outputs/documentation/index.html
• Table pages: ./outputs/documentation/tables/*.html
• Schema helper: ./outputs/schema/schema.py

schema.py is always emitted. If Spark is unavailable locally, run schema.py in a Spark environment, save the resulting schema.csv to ./outputs/schema/schema.csv, and run bolt generate documentation again.

layers

Creates one executable script per layer:

• Layer scripts: ./outputs/layers/<layer>.py

Each file includes Spark session setup, ETL base code, and the jobs for that layer in dependency aware order.

notebook

Creates a notebook with setup cells plus generated job cells:

• Notebook: ./outputs/notebook/etl_jobs_notebook.ipynb

all

Runs all four generators in one command: airflow, documentation, layers, and notebook.

How dependency resolution works

Generation is layer scoped and queue based. The resolver checks each job’s input_tables and delays jobs that depend on unfinished jobs from the same layer.

In practice, this gives you a topological style order inside each layer:

1. Start with jobs that have no same layer dependencies.
2. Mark each completed job as available.
3. Revisit delayed jobs until they become runnable.

Cross layer inputs such as bronze_customers used by a silver job are treated as external to the current layer pass and do not block ordering inside that layer.

If dependency resolution stalls because of a cycle or bad references, behavior differs by generator:

• layers raises an error about circular dependencies.
• airflow prints a warning that some jobs have circular dependencies.
• documentation and notebook stop making progress after safety passes, so unresolved jobs are not emitted.

Typical workflow

Use this sequence when you want fresh assets after config or job changes:

bolt generate layers
bolt generate notebook
bolt generate documentation
bolt generate airflow

For routine refreshes, bolt generate all is the short path.