Spark session profiles

After this page, you will know how Spark session profiles are resolved in bolt-pipeliner==0.2.6. You will also know which profiles work today and how to override profile selection with an environment variable.

What a profile controls

Spark profiles provide two things:

1. The runtime target name, read from [runtime].target in a TOML file.
2. The Spark config map, read from [spark] in that same file.

bolt run resolves a profile, then calls create_session(profile_name, spark_config=...).

Profile support in 0.2.6

The interactive bolt init prompt offers these Spark profile names:

• local
• databricks
• emr
• glue
• gcp
• azure
• k8s

In the installed 0.2.6 package, only local has an implementation module under bolt_pipeliner/sessions/.

• Implemented: local via bolt_pipeliner.sessions.local
• Not implemented as session modules: databricks, emr, glue, gcp, azure, k8s

If you force an unimplemented profile, Python raises ModuleNotFoundError when create_session tries to import bolt_pipeliner.sessions.<profile>.

Profile resolution order

resolve_spark_profile uses this priority order:

1. BOLT_SPARK_PROFILE
2. configs.spark_profile in configs/etl_config.yaml
3. configs/spark/local.toml if present
4. The only configs/spark/*.toml file, when exactly one exists
5. Fallback to profile local with empty Spark config

If the selected TOML file has [runtime].target, that value becomes the final profile name used by create_session.

File layout and examples

Set a default profile in configs/etl_config.yaml:

configs:
  spark_profile: local

Create configs/spark/local.toml:

[runtime]
target = "local"
 
[spark]
"spark.sql.shuffle.partitions" = 200
"spark.serializer" = "org.apache.spark.serializer.KryoSerializer"

When you pick a non local profile during bolt init, the scaffold writes a placeholder TOML with a TODO marker:

[runtime]
target = "databricks"
 
# TODO: configure databricks Spark profile. See bolt_pipeliner/sessions/databricks.py.
 
[spark]

In 0.2.6, that referenced sessions/databricks.py module is not shipped.

Override with BOLT_SPARK_PROFILE

Use BOLT_SPARK_PROFILE to switch profiles without editing YAML.

BOLT_SPARK_PROFILE=local bolt run --verbose

This variable has higher priority than configs.spark_profile.

Practical guidance for 0.2.6

Use local for real runs with Spark in this release. Treat other profile names as scaffold placeholders until matching bolt_pipeliner/sessions/<profile>.py modules exist.