Bolt Pipeliner Overview

bolt-pipeliner is a Python ETL framework for building layered data pipelines with a small amount of code. After this page, you will know what problem it solves and when it is a good fit.

The framework is config driven and dbt inspired. It keeps the workflow in Python modules, so teams that already use Python for transforms can move quickly.

What it solves

Most pipeline projects repeat the same plumbing work. You still need a clear layer model, dependency ordering, test checks, and delivery artifacts.

bolt-pipeliner gives you one consistent runtime for that work:

• Layered execution from flatfile through bronze, silver, gold, and diamond.
• A single etl_config.yaml file that declares jobs, dependencies, and runtime settings.
• A single bolt CLI for project setup, runs, tests, and code generation.

The core idea: one YAML and one CLI

Define jobs in configuration, then keep each transformation in a focused Python module.

configs:
  flatfile_location: ./data
  output_location: ./outputs
 
layers:
  flatfile: jobs/flatfile
  bronze: jobs/bronze
 
flatfile:
  - module: ingest_orders
    input_tables: []
    output_table_name: orders_raw
 
bronze:
  - module: clean_orders
    input_tables: [orders_raw]
    output_table_name: orders_clean

Run the same project lifecycle with the CLI:

bolt init my_project
bolt run --verbose
bolt test
bolt generate documentation

When to choose Bolt Pipeliner

Choose it when you want dbt-like structure but your team prefers Python over SQL first tooling. It works well when you want a clear layer model, simple configuration, and optional generators for documentation, Airflow, and notebooks.

If your team already has strong Python data engineering patterns and wants a lightweight framework instead of a large platform, bolt-pipeliner is a practical choice.