Core concepts

This page gives you the mental model for how bolt-pipeliner executes work. After reading it, you can explain layers, engine selection, and why the framework keeps most decisions in config.

Think in layers

bolt-pipeliner executes jobs in declared layer order. The default architecture moves data through these stages:

• flatfile: ingestion from local or remote flat files into managed tables.
• bronze: raw but structured tables, usually close to source shape.
• silver: cleaned and standardized tables for reuse.
• gold: business ready marts for analytics and reporting.
• diamond: advanced outputs such as feature tables or model artifacts.

Each job declares input_tables and an output_table_name. The runner executes jobs in the layer order from your YAML, then job order inside each layer.

Config is the control plane

The framework is config-driven. Your etl_config.yaml controls runtime behavior while job modules focus on transformation logic.

In practice, etl_config.yaml defines:

• global runtime settings in configs
• where each layer lives in layers
• each layer job list with module, input_tables, and output_table_name

configs:
  flatfile_location: ./data
  output_location: ./outputs
 
layers:
  flatfile: etl/flatfile
  bronze: etl/bronze
  silver: etl/silver
 
flatfile:
  - module: ingest_orders
    input_tables: []
    output_table_name: orders_raw
 
bronze:
  - module: clean_orders
    input_tables: [orders_raw]
    output_table_name: orders_clean
    class_name: ETLBaseParquetPandas

Engine choice comes from the base class

You pick the execution engine per job through class_name. In 0.2.6, these built in options are available:

• ETLBase: Spark with Iceberg
• ETLBaseDelta: Spark with Delta
• ETLBaseParquet: Spark with Parquet
• ETLBaseParquetPandas: Pandas with Parquet
• ETLBaseParquetPolars: Polars with Parquet

This lets one project mix engines when needed, while keeping the same layer model and CLI workflow.

Lazy imports keep startup lightweight

Engine imports happen only when they are needed:

• bolt run creates a Spark session only if no session was passed to run(...).
• base classes are imported at runtime when each job is resolved.
• package import does not eagerly load Spark, Pandas, or Polars base modules.

This design helps local development stay fast and allows lightweight projects to run without pulling the full Spark stack into every import path.