Configuration

After this page, you will be able to write a valid etl_config.yaml from scratch and understand which keys are required. You will also know the defaults that bolt-pipeliner==0.2.6 applies when keys are missing.

All keys and defaults here are verified against the installed package code in bolt_pipeliner/config/loader.py and bolt_pipeliner/runner.py.

File structure at a glance

etl_config.yaml has three parts:

• configs: global runtime settings
• layers: map each layer name to its module directory
• One list per layer: each list item is a job definition

Minimal shape:

configs: {}
 
layers:
  flatfile: etl/_flatfile
  bronze: etl/0_bronze
  silver: etl/1_silver
  gold: etl/2_gold
 
flatfile: []
bronze: []
silver: []
gold: []

Annotated example

This example follows the local Pandas preset pattern and can run on a single machine.

configs:
  output_location: "data/layers"      # where non-flatfile outputs are written
  flatfile_location: "data/flatfiles" # where flatfile inputs are read from
  schema: sample_project
  catalog: dev_catalog
  incremental_column: year_month
  incremental_type: int
  incremental_unit: 3
  incremental_date_grain: monthly
 
layers:
  flatfile: etl/_flatfile
  bronze: etl/0_bronze
  silver: etl/1_silver
  gold: etl/2_gold
 
flatfile:
  - module: flatfile_example
    class_name: ETLBaseParquetPandas
    description: Load source events from CSV.
    input_tables:
      events: "example.csv"
    output_table_name: events
 
bronze:
  - module: bronze_example
    class_name: ETLBaseParquetPandas
    input_tables:
      events: flatfile_events
    output_table_name: events_clean
    partition_by: [year_month]
    incremental: true
    incremental_column: year_month
    incremental_type: int
    incremental_unit: append
    tests:
      - not_null: [event_id, year_month]
      - row_count: {min: 1}
 
silver:
  - module: silver_example
    class_name: ETLBaseParquetPandas
    input_tables:
      events: bronze_events_clean
    output_table_name: events_features
 
gold:
  - module: gold_example
    class_name: ETLBaseParquetPandas
    input_tables:
      events: silver_events_features
    output_table_name: events_mart

Run it with:

bolt run --config configs/etl_config.yaml --verbose

configs block

The loader in 0.2.6 fills these defaults when they are missing:

• schema: cxdw_dm
• incremental_column: year_month
• incremental_type: int
• incremental_unit: 3
• incremental_date_grain: monthly
• flatfile_location: empty string by default
• output_location: empty string by default

catalog is read by the runner and defaults to dev_catalog at runtime.

Storage location aliases

Older configs may use:

• flatfile_bucket
• output_bucket

In 0.2.6, these legacy keys are still accepted. They are normalized to flatfile_location and output_location.

layers block

layers maps logical layer names to module directories. Example:

layers:
  flatfile: etl/_flatfile
  bronze: etl/0_bronze
  silver: etl/1_silver
  gold: etl/2_gold
  diamond: etl/3_diamond

For each key in layers, define a top level list with the same name. If a layer has no jobs yet, use an empty list.

Per job keys

Required keys for execution:

• module
• input_tables
• output_table_name

Optional keys used by runtime and generators:

• class_name (default: ETLBase)
• partition_by
• unload (default: true)
• incremental (default: false in runner)
• incremental_column
• incremental_type
• incremental_unit
• incremental_date_grain
• description
• tests

Backward compatibility note: _input_tables is automatically renamed to input_tables.

Supported flatfile formats

Format detection is extension based. These extensions are supported:

• CSV: .csv
• Parquet: .parquet
• Excel family: .xlsx, .xls, .xlsm, .xlsb, .ods, .odf, .odt
• JSON document: .json
• JSON Lines: .jsonl, .ndjson

If an extension is not recognized, the run fails with an unsupported format error.