Securely Managing Secrets for Spark Jobs

Security is critical when handling sensitive data like passwords, API keys, and connection strings. Hardcoding credentials in your job code or Docker images is a major security risk. Instead, إيلوم leverages native Kubernetes Secrets to inject credentials securely at runtime.

Step 1: Creating Kubernetes Secrets for Spark​

First, you need to create a Kubernetes Secret object. This secret must reside in the same namespace where your Ilum Job will be executed (typically افتراضي or your specific tenant namespace).

kubectl create secret generic my-db-creds \
  --from-literal=username=admin \
  --from-literal=password=SuperSecret123 \
  -n default

Step 2: Mounting Secrets in Ilum Spark Jobs​

You can expose the secret to your Spark driver and executors in two ways: as environment variables or as files via volume mounts.

Method A: Environment Variables (Recommended)​

This is the most common method for passing simple keys like database passwords or API tokens.

# Valid for Spark on Kubernetes
# Syntax: spark.kubernetes.[driver|executor].secretKeyRef.[ENV_VAR_NAME]=name-of-secret:key-in-secret

spark.kubernetes.driver.secretKeyRef.DB_PASSWORD= my-db-creds:password
spark.kubernetes.executor.secretKeyRef.DB_PASSWORD= my-db-creds:password
spark.kubernetes.driver.secretKeyRef.DB_USER= my-db-creds:username
spark.kubernetes.executor.secretKeyRef.DB_USER= my-db-creds:username

استورد  os

# Retrieve credentials securely from environment
db_user =  os. environ. حصل ( "DB_USER") 
db_pass =  os. environ. حصل ( "DB_PASSWORD") 

لو  لا  db_pass: 
    raise ValueError( "Database password not found in environment variables") 

طبع ( f"Connecting as user: { db_user} " )  # Safe to log username, never log password!

val dbUser =  System. getenv( "DB_USER") 
val dbPass =  System. getenv( "DB_PASSWORD") 

Method B: Volume Mounts​

Use this method when your application expects a file path (e.g., SSL certificates, Keytab files, or configuration files).

# Mount secret 'my-db-creds' to /etc/secrets/db
spark.kubernetes.driver.secrets.my-db-creds= /etc/secrets/db
spark.kubernetes.executor.secrets.my-db-creds= /etc/secrets/db

# Read password from the mounted file
مع  open( '/etc/secrets/db/password',  'r')  مثل f : 
    db_pass = f . قرأ ( ) . strip( ) 

مع  open( '/etc/secrets/db/username',  'r')  مثل f : 
    db_user = f . قرأ ( ) . strip( ) 

Best Practices: Env Vars vs. Volumes​

Step 3: Using Secrets in Airflow DAGs​

If you are orchestrating jobs with the Ilum Airflow integration, you can define these configurations directly in your DAGs using the IlumSparkSubmitOperator.

من  airflow استورد  DAG
من إيلوم . airflow. operators استورد  IlumSparkSubmitOperator

# ... (DAG definition)

submit_spark_job =  IlumSparkSubmitOperator( 
    task_id= "secure_spark_job", 
    spark_conf= { 
        "spark.kubernetes.driver.secretKeyRef.DB_PASSWORD":  "my-db-creds:password", 
        "spark.kubernetes.executor.secretKeyRef.DB_PASSWORD":  "my-db-creds:password"
    } 
    # ... other configurations
) 

Troubleshooting & Verification​

verifying Secret Existence​

Before running your job, verify the secret exists and contains the correct data:

# List secrets in the namespace
kubectl get secrets -n default

# Decode secret values (for debugging only!)
kubectl get secret my-db-creds -o jsonpath='{.data.password}' -n default | base64 --decode

Debugging Running Pods​

If your job fails with authentication errors, inspect the running pod to ensure variables are mounted correctly.

# 1. Find the driver pod name
kubectl get pods -n default | grep spark-driver

# 2. Check environment variables inside the pod
kubectl exec -it <spark-driver-pod-name> -n default -- env | grep DB_

Frequently Asked Questions (FAQ)​