Macros

After this page, you will be able to create reusable helpers in macros/ and call them from ETL jobs. You will also understand why macros in bolt-pipeliner==0.2.6 are plain Python with normal imports.

What macros are in bolt-pipeliner

In this framework, a macro is just a Python function you keep in your project.

bolt init scaffolds a macros/ package with this docstring:

"""Project-local reusable transforms. Import from your jobs as
`from macros.dates import month_floor` etc.
"""

There is no macro registry, no decorator contract, and no template DSL.

Why regular imports work

When you run bolt run, the runner inserts your project root into sys.path before importing job modules. This means a job can import from macros the same way it imports any other local Python package.

Example macro module

Create macros/dates.py:

import pandas as pd
 
 
def month_floor(ts_series: pd.Series) -> pd.Series:
    return pd.to_datetime(ts_series).dt.to_period("M").dt.to_timestamp()

Optional convenience export in macros/__init__.py:

from .dates import month_floor

Use the macro in a job

etl/1_silver/silver_orders.py:

import pandas as pd
 
from macros.dates import month_floor
 
 
def process_data(self, input_tables):
    orders = input_tables["orders"].copy()
    orders["order_month"] = month_floor(orders["order_ts"])
 
    out = (
        orders.groupby(["customer_id", "order_month"], as_index=False)["amount"]
        .sum()
        .rename(columns={"amount": "gross_amount"})
    )
    return out

Matching config entry:

silver:
  - module: silver_orders
    class_name: ETLBaseParquetPandas
    input_tables:
      orders: bronze_orders
    output_table_name: customer_monthly_sales

This runs with the same bolt run flow as any other job.

Design point to remember

Macros are only regular Python modules inside your repo. If you can import it in Python, you can use it in a job.