Data quality tests

After this page, you will be able to declare tests in etl_config.yaml and run them with bolt test. You will also know how pass or fail status is exposed for CI and notebooks.

All behavior on this page is verified against bolt-pipeliner==0.2.6.

Where tests live

Tests are declared per job under the tests: key.

silver:
  - module: silver_customer_features
    class_name: ETLBaseParquetPandas
    input_tables:
      customers: bronze_customers
    output_table_name: customer_features
    tests:
      - not_null: [customer_id, year_month]
      - unique: [customer_id, year_month]
      - row_count: {min: 1, max: 1000000}
      - schema: [customer_id, year_month, segment]
      - freshness: {column: year_month, max_age_days: 90}

Each list item is one check spec. Use either a one key mapping with parameters, or a plain string when a check accepts defaults.

Run tests

bolt test [--config PATH] [--layer LAYER] [--module MODULE]

Common patterns:

# Run all tests in the project config.
bolt test
 
# Run only one layer.
bolt test --layer silver
 
# Run one module in one layer.
bolt test --layer silver --module customer_features

bolt test returns exit code 0 when all checks pass. It returns exit code 1 when any check fails, which is the behavior you want for CI pipelines.

Passing run with PASS banners:

Failing run with a FAIL banner:

Built-in checks

bolt-pipeliner==0.2.6 supports five checks.

not_null

YAML form:

- not_null: [col_a, col_b]

Parameters passed to the check function:

• columns: list of columns that must be non null

The check fails if any listed column is missing, or if any row has a null in those columns.

unique

YAML form:

- unique: [col_a, col_b]

Parameters passed to the check function:

• columns: list of columns that define a composite unique key

The check fails when duplicate key combinations exist.

row_count

YAML form:

- row_count: {min: 1, max: 1000000}

Parameters passed to the check function:

• min: minimum allowed row count, default is 1
• max: optional maximum allowed row count

The check fails if the DataFrame row count falls outside the allowed range.

schema

YAML form:

- schema: [customer_id, year_month, segment]

Parameters passed to the check function:

• expected: list of required columns

The check validates presence of expected columns. Extra columns are allowed.

freshness

YAML form:

- freshness: {column: year_month, max_age_days: 90}

Parameters passed to the check function:

• column: field used to evaluate recency
• max_age_days: maximum allowed age in days

The check accepts either:

• a YYYYMM integer style value in column
• a date or datetime style value in column

It fails when the latest value in the column is older than max_age_days.

Parameter coercion rules

The test runner normalizes shorthand YAML into function parameters:

• not_null: [a, b] becomes columns=[a, b]
• unique: [a, b] becomes columns=[a, b]
• schema: [a, b] becomes expected=[a, b]
• row_count: {min: 1, max: 10} is passed through as is
• freshness: {column: year_month, max_age_days: 90} is passed through as is

Notebook and HTML rendering

Each check returns a TestResult object with PASS or FAIL status, details text, and optional failed row count.

In notebook contexts, TestResult implements _repr_html_(), so each result renders as inline HTML with colored pass or fail labels.

The testing runner also includes render_html(results), which joins that HTML output into one block for display.