ML training

After this page, you will know where ML code lives in a bolt-pipeliner==0.2.6 project and how to productionize notebook experiments. You will also know how to run the same flow with or without MLflow.

All details here are verified from the installed package at .venv/lib/python3.11/site-packages/bolt_pipeliner/cli/init.py and a real scaffold generated with bolt init --preset diamond.

What bolt init scaffolds for ML

When you enable ML scaffolding, Bolt creates two directories side by side:

• models/: production Python modules for training and loading models
• model_notebooks/: notebook workspace for feature exploration, tuning, and evaluation

The generated structure looks like this:

models/
  train_example.py
model_notebooks/
  README.md
  train_example.ipynb

In 0.2.6 there is no dedicated bolt train command. Training is regular Python code that you call from your ETL jobs.

Why the diamond layer is the default home

Bolt treats ML jobs as downstream consumers of curated data from gold. The init flow explicitly calls diamond the conventional place for model artifacts and prediction outputs.

If you opt into ML in the interactive wizard and your selected architecture does not include diamond, Bolt asks whether it should add that layer.

Example from a generated config:

layers:
  flatfile: etl/_flatfile
  bronze: etl/0_bronze
  silver: etl/1_silver
  gold: etl/2_gold
  diamond: etl/3_diamond
 
diamond: []

Typical workflow

Use this sequence:

1. Build curated features in gold jobs.
2. Experiment in model_notebooks/train_example.ipynb.
3. Move stable training and loading logic to models/train_example.py.
4. Call that logic from a diamond job so training or scoring runs on your pipeline schedule.

Path 1: with MLflow (recommended)

The generated files treat MLflow as recommended for experiment tracking and model registry. They include commented imports and example calls, but they do not require MLflow to be installed.

You can follow this pattern:

pip install mlflow
export MLFLOW_TRACKING_URI=sqlite:///mlflow.db
mlflow ui

Then in your training code:

import mlflow
import mlflow.sklearn
 
 
def train_and_log(model, auc):
    with mlflow.start_run():
        mlflow.log_metric("auc", auc)
        mlflow.sklearn.log_model(
            model,
            artifact_path="model",
            registered_model_name="example_model",
        )

For inference in a diamond job, load from the registry inside models/train_example.py and keep ETL orchestration code separate from model registry details.

Path 2: without MLflow

MLflow is optional in 0.2.6. The scaffolded modules stay valid Python with MLflow imports commented out.

If you do not want MLflow, store serialized models in your own versioned path and load by explicit version.

from pathlib import Path
import pickle
 
 
def save_model(model, version: str, root: str = "artifacts"):
    out = Path(root) / version / "model.pkl"
    out.parent.mkdir(parents=True, exist_ok=True)
    with out.open("wb") as f:
        pickle.dump(model, f)
    return out
 
 
def load_model(version: str, root: str = "artifacts"):
    model_path = Path(root) / version / "model.pkl"
    with model_path.open("rb") as f:
        return pickle.load(f)

This keeps your project dependency light while preserving a clear promotion path from notebook to production module to diamond ETL execution.