GitOps for Spark on Kubernetes using ArgoCD and Ilum

GitOps is an operational model where the state of your Kubernetes cluster is managed entirely via a Git repository. ArgoCD is the industry standard for implementing GitOps.

What is GitOps with Ilum?​

Using GitOps with Ilum allows you to manage your شرارة على Kubernetes infrastructure declaratively. By pairing ArgoCD with Ilum, you achieve automated syncs, drift detection, and version-controlled cluster configuration. This approach ensures that your data platform's state always matches your source code.

Why use GitOps for Ilum?​

• Audit Trails: Every change to your Spark platform is committed to Git, telling you exactly who changed what and when.
• Disaster Recovery: Restore your entire cluster state from Git instantly in case of failure.
• Consistency: Prevent configuration drift between Development, Staging, and Production environments.

This guide explains how to deploy and manage Ilum instances using an App-of-Apps pattern with ArgoCD.

المتطلبات المسبقه ​

• A running Kubernetes cluster (see Installation Guide).
• ArgoCD installed in the argocd namespace.
• A Git repository to store your configuration (e.g., https://github.com/my-org/ilum-gitops).
• Basic understanding of Ilum Architecture.

1. Directory Structure​

Organize your GitOps repository to separate the Helm chart reference from the environment-specific values.

ilum-gitops/
├── apps/
│   └── ilum-prod.yaml        # ArgoCD Application manifest
└── values/
    └── values-prod.yaml      # Environment-specific overrides

2. Defining the Environment Configuration​

إنشاء values-prod.yaml file. This file contains only the overrides needed for your production environment, keeping the configuration clean.

# Production overrides for Ilum

# Disable development features
ilum-jupyter: 
  تمكين :  خطأ 

جيتا : 
  تمكين :  خطأ 

# Use external database (Recommended for Prod)
postgresql: 
  تمكين :  خطأ 

# Enable Monitoring
إيلوم كور : 
  job: 
    prometheus: 
      تمكين :  صحيح 

3. Creating the ArgoCD Application​

Create the Application manifest that tells ArgoCD how to combine the official Ilum Helm Chart with your custom values.

apiVersion :  argoproj.io/v1alpha1
نوع : تطبيق 
البيانات الوصفية : 
  اسم : إيلوم - همز 
  Namespace :  argocd
المواصفات : 
  مشروع : افتراضي 
  
  # Destination Cluster
  destination: 
    ملقم : https : //kubernetes.default.svc
    Namespace : إيلوم - همز 

  # Source Configuration
  sources: 
    # 1. Official Ilum Helm Chart
    -  chart: إيلوم 
      repoURL: https : //charts.ilum.cloud
      targetRevision:  6.6.1
      helm: 
        valueFiles: 
        -  $values/values/values- prod.yaml  # Reference to the second source

    # 2. Your GitOps Repository (contains the values file)
    -  repoURL: https : //github.com/my- org/ilum- gitops
      targetRevision: رأس 
      الرقم المرجعي : القيم 

  # Sync Policy
  syncPolicy: 
    automated: 
      prune:  صحيح       # Delete resources removed from Git
      selfHeal:  صحيح    # Auto-correct drift
    syncOptions: 
      -  CreateNamespace=true

4. Deploying​

Apply the Application manifest to your cluster:

kubectl apply -f apps/ilum-prod.yaml -n argocd

Verification​

1. ArgoCD UI: Navigate to the ArgoCD dashboard. You should see the ilum-prod application syncing.
2. Kubectl: Check the resources in the ilum-prod namespace.

kubectl get pods -n ilum-prod

5. Workflow: Updating Configuration​

To change a configuration (e.g., enable monitoring or change resource limits), you no longer run ترقية Helm . Instead:

1. حرر values/values-prod.yaml in your Git repository.
2. Commit and Push the changes.
3. ArgoCD detects the commit and automatically syncs the cluster state to match.

This provides a full audit trail of every change made to your production infrastructure.

Frequently Asked Questions (FAQ)​