تخطي إلى المحتوى الرئيسي

CI/CD for Ilum Interactive Services with GitHub Actions

This guide demonstrates how to automate the deployment of Ilum Interactive Services استخدام GitHub Actions with self-hosted runners running directly on your Kubernetes cluster. By combining the Ilum REST API with a CI/CD pipeline, you can deploy and update services automatically on every push or merge.

Quick Summary
  • Self-Hosted Runners: Deploy GitHub Actions Runner Controller (ARC) on Kubernetes so workflows execute inside your cluster with direct access to Ilum.
  • Service Code in Git: Store your Ilum service implementation (service.py) in a GitHub repository.
  • Automated Deployment: A GitHub Actions workflow creates/updates Ilum service groups via the REST API on every push to رئيسي.
  • Update via Pull Requests: Modify service logic in a feature branch, open a PR, merge — and the service is automatically redeployed with the new code.

المتطلبات المسبقه​

  • A running Ilum instance on Kubernetes (with all pods healthy)
  • كوبيكتل و helm configured for your cluster
  • A GitHub account with permissions to create repositories and personal access tokens
  • cert-manager installed on the cluster

Step 1: Install cert-manager​

Actions Runner Controller requires cert-manager for TLS certificate management. Install it if not already present:

Install cert-manager
kubectl apply -و https://github.com/cert-manager/cert-manager/releases/download/v1.16.2/cert-manager.yaml

Wait for all cert-manager pods to be ready:

Wait for cert-manager
كوبيكتل wait --for=condition=Ready pods --all -ن cert-manager --timeout=120s

Step 2: Create a GitHub Personal Access Token​

The self-hosted runner needs a GitHub token to register itself with your repository.

  1. Navigate to GitHub → Settings → Developer Settings → Personal access tokens → Tokens (classic).
  2. نقر Generate new token (classic).
  3. Set a descriptive name (e.g., ServiceManagementToken).
  4. Select the repo scope (full control of private repositories).
  5. نقر Generate token and copy the value immediately — you won't see it again.

Step 3: Install Actions Runner Controller (ARC)​

3.1 Add the Helm Repository​

Add ARC Helm Repo
helm repo جمع actions-runner-controller \
  https://actions-runner-controller.github.io/actions-runner-controller

3.2 Create the Namespace and Secret​

Create Namespace and Secret
# Create namespace for runners
kubectl create ns arc-runners

# Store the GitHub token as a Kubernetes secret
kubectl create secret generic github-token \
  --namespace arc-runners \
  --from-literal=github_token=<YOUR_GITHUB_TOKEN>

3.3 Install ARC via Helm​

Install ARC
helm install arc \
  actions-runner-controller/actions-runner-controller \
  --namespace arc-runners \
  --جبر authSecret.create=خطأ \
  --جبر authSecret.name=github-token

Step 4: Create a Runner Deployment​

إنشاء runner-deployment.yaml file that tells ARC to spawn a self-hosted runner for your repository:

runner-deployment.yaml
apiVersion: actions.summerwind.dev/v1alpha1
نوع: RunnerDeployment
البيانات الوصفية:
  اسم: إيلوم-demo-runner
  Namespace: arc-runners
المواصفات:
  النسخ المتماثله: 1
  قالب:
    المواصفات:
      مستودع: < الخاص بك-org>/IlumServiceManagment
      تسميات:
        - ذات-hosted
        - linux

Apply it:

Deploy Runner
kubectl apply -و runner-deployment.yaml

The runner pod will register itself with GitHub and appear in your repository's Settings → Actions → Runners as an available self-hosted runner.

Step 5: Create the Service Repository​

Create a new GitHub repository (e.g., IlumServiceManagment) with the following structure:

IlumServiceManagment/
├── service.py                      # Ilum service implementation
└── .github/
    └── workflows/
        └── deploy.yml              # GitHub Actions workflow

5.1 Service Implementation (service.py)​

This file contains the Ilum Interactive Service code. The service extends Ilum Job and implements the ركض method:

service.py
من إيلوم.واجهة برمجة التطبيقات استورد Ilum Job
من بايسبارك.SQL.functions استورد col, sum مثل spark_sum


فصل SparkInteractiveExample(Ilum Job):
    مواطنه ركض(ذات, شراره, التكوين) -> str:
        table_name = التكوين.حصل("الجدول")
        database_name = التكوين.حصل('database')
        report_lines = []

        لو لا table_name:
            raise ValueError("Config must provide a 'table' key")

        لو database_name:
            شراره.كتالوج.setCurrentDatabase(database_name)
            report_lines.append(f"Using database: {database_name}")

        لو table_name لا في [t.اسم من أجل t في شراره.كتالوج.listTables()]:
            raise ValueError(f"Table '{table_name}' not found in catalog")

        مدافع = شراره.جدول(table_name)

        report_lines.append(f"=== Details for table: {table_name} ===")

        total_rows = مدافع.عد()
        report_lines.append(f"Total rows: {total_rows}")

        total_columns = len(مدافع.columns)
        report_lines.append(f"Total columns: {total_columns}")

        report_lines.append("Distinct values per column:")
        من أجل c في مدافع.columns:
            distinct_count = مدافع.select(c).distinct().عد()
            report_lines.append(f"  {c}: {distinct_count}")

        report_lines.append("Schema:")
        من أجل f في مدافع.مخطط.fields:
            report_lines.append(f"  {f.اسم}: {f.dataType}")

        report_lines.append("Sample data (first 5 rows):")
        sample_rows = مدافع.take(5)
        من أجل صف في sample_rows:
            report_lines.append(str(صف.asDict()))

        report_lines.append("Null counts per column:")
        null_counts_df = مدافع.select(
            [spark_sum(col(c).isNull().cast("int")).alias(c) من أجل c في مدافع.columns]
        )
        null_counts = null_counts_df.collect()[0].asDict()
        من أجل c, v في null_counts.items():
            report_lines.append(f"  {c}: {v}")

        أعاد "\n".join(report_lines)

5.2 GitHub Actions Workflow (.github/workflows/deploy.yml)​

This workflow runs on every push to رئيسي and uses the Ilum REST API to deploy the service:

.github/workflows/deploy.yml
اسم: Deploy ILUM Group

on:
  دفع:
    branches: [رئيسي]

الحياه الفطريه:
  ILUM_DEMO_SERVICE: ILUM_DEMO_SERVICE

jobs:
  deploy:
    runs-on: ذات-hosted
    steps:
      - اسم: Checkout code
        يستخدم: actions/checkout@v4

      - اسم: Check if group ILUM_DEMO_SERVICE exists
        معرف: check_group
        ركض: |
          echo "Checking if group ILUM_DEMO_SERVICE exists..."

          RESPONSE=$(curl -s -X GET \
            -w "\nHTTP_STATUS:%{http_code}" \
            "http://ilum-core.ilum.svc.cluster.local:9888/api/v1/group?name=$ILUM_DEMO_SERVICE")

          HTTP_STATUS=$(echo "$RESPONSE" | grep HTTP_STATUS | cut -d':' -f2)
          BODY=$(echo "$RESPONSE" | sed '/HTTP_STATUS/d')

          GROUP_ID=$(echo "$BODY" | jq -r '.content[0].groupId // empty')

          لو [ -n "$GROUP_ID" ] && [ "$GROUP_ID" != "null" ]; then
            echo "Group ILUM_DEMO_SERVICE found with ID: $GROUP_ID"
            echo "GROUP_ID=$GROUP_ID" >> $GITHUB_OUTPUT
            echo "GROUP_EXISTS=true" >> $GITHUB_OUTPUT
          اخر
            echo "Group ILUM_DEMO_SERVICE does not exist."
            echo "GROUP_EXISTS=false" >> $GITHUB_OUTPUT
          fi

      - اسم: Delete existing group
        لو: steps.check_group.outputs.GROUP_EXISTS == 'true'
        ركض: |
          GROUP_ID="${{ steps.check_group.outputs.GROUP_ID }}"
          echo "Stopping and deleting group ILUM_DEMO_SERVICE (ID: $GROUP_ID)..."

          حليقه -s -X POST \
            HTTP://ilum-core.ilum.svc.cluster.local:9888/api/v1/group/$GROUP_ID/stop

          RESPONSE=$(curl -s -X DELETE \
            -w "\nHTTP_STATUS:%{http_code}" \
            HTTP://ilum-core.ilum.svc.cluster.local:9888/api/v1/group/$GROUP_ID)

          HTTP_STATUS=$(echo "$RESPONSE" | grep HTTP_STATUS | cut -d':' -f2)

          لو [ "$HTTP_STATUS" -ne 200 ]; then
            echo "Error: Failed to delete group (Status: $HTTP_STATUS)"
            exit 1
          fi

          echo "Group ILUM_DEMO_SERVICE deleted successfully."

      - اسم: Create new group
        ركض: |
          echo "Creating group ILUM_DEMO_SERVICE with service.py..."

          RESPONSE=$(curl -s -X POST \
            -F "name=ILUM_DEMO_SERVICE" \
            -F "[البريد الإلكتروني محمي]" \
            -F "clusterName=default" \
            -F "language=PYTHON" \
            -w "\nHTTP_STATUS:%{http_code}" \
            HTTP://ilum-core.ilum.svc.cluster.local:9888/api/v1/group)

          HTTP_STATUS=$(echo "$RESPONSE" | grep HTTP_STATUS | cut -d':' -f2)
          BODY=$(echo "$RESPONSE" | sed '/HTTP_STATUS/d')

          echo "HTTP Status: $HTTP_STATUS"
          echo "Response Body: $BODY"

          لو [ "$HTTP_STATUS" -ne 200 ]; then
            echo "Error: Failed to create group (Status: $HTTP_STATUS)"
            exit 1
          fi

          GROUP_ID=$(echo "$BODY" | jq -r '.groupId // empty')
          echo "Group ILUM_DEMO_SERVICE created successfully with ID: $GROUP_ID"

Commit both files to the repository.

How the Workflow Works

Since the runner pod runs inside the Kubernetes cluster, it can directly access the Ilum Core API via the internal service DNS (ilum-core.ilum.svc.cluster.local:9888). The workflow:

  1. Checks if a service group named ILUM_DEMO_SERVICE already exists.
  2. Deletes the existing group (stops it first, then removes it).
  3. Creates a new group by uploading service.py via the REST API.

Step 6: Verify the Deployment​

Once the workflow completes successfully (visible in the Actions tab on GitHub), navigate to the Ilum UI:

  1. الانتقال إلى خدمات in the sidebar.
  2. You should see the ILUM_DEMO_SERVICE service group with status Active.
  3. Click on it and go to the Execute Job التبويب.
  4. Set the فصل ل service.SparkInteractiveExample.
  5. Provide parameters as JSON:
Execution Parameters
{
    "database": "ilum_example_product_sales",
    "table": "products"
}
  1. نقر أعدم. The result will display table details including row count, schema, sample data, and null counts.

Step 7: Update the Service via Pull Request​

The key benefit of this CI/CD approach is that updating your service is as simple as modifying code and merging a pull request.

Example: Adding Table Listing Feature​

  1. Create a new branch (e.g., feature/list-tables) from رئيسي.
  2. حرر service.py to add functionality — for example, listing other tables in the database:
Added to service.py (at the end of run method)
        # List other tables in the current database
        current_db = شراره.كتالوج.currentDatabase()
        report_lines.append(f"Tables in database '{current_db}':")
        tables = شراره.كتالوج.listTables(current_db)
        other_tables = [t.اسم من أجل t في tables لو t.اسم != table_name]
        لو other_tables:
            من أجل tname في other_tables:
                report_lines.append(f"  {tname}")
        اخر:
            report_lines.append("  (no other tables)")

        أعاد "\n".join(report_lines)
  1. Commit the change and open a Pull Request من feature/list-tables into رئيسي.
  2. Merge the PR.
  3. The GitHub Actions workflow triggers automatically, stops the old service, and deploys the updated version.
  4. Execute the job again in Ilum — the result now includes the new "Tables in database" section.

استكشاف الاخطاء​

Runner pod not registering with GitHub
  • Verify the GitHub token secret is correct: kubectl get secret github-token -n arc-runners -o yaml
  • Check the runner pod logs: kubectl logs -n arc-runners -l app=ilum-demo-runner
  • تأكد من أن مستودع field in runner-deployment.yaml matches your GitHub repository exactly (including organization/user).
Workflow fails with connection refused to ilum-core

The self-hosted runner must be in the same Kubernetes cluster as Ilum. Verify:

  • The Ilum namespace and service name are correct. Use kubectl get svc -n ilum to find the correct service DNS.
  • The runner pod has network access to the Ilum namespace. Check for NetworkPolicies that may block cross-namespace traffic.
Service group creation returns HTTP 400
  • تأكد من أن service.py file contains a valid class extending Ilum Job.
  • Check that clusterName matches an existing Ilum cluster (usually افتراضي).
  • Verify the file is being uploaded correctly with -F "[البريد الإلكتروني محمي]".
Job execution returns "Table not found in catalog"

Make sure the database و جدول parameters match existing data in your Ilum environment. You can check available databases and tables via the SQL module or a Jupyter notebook.

Frequently Asked Questions (FAQ)​

Why use self-hosted runners instead of GitHub-hosted runners?

GitHub-hosted runners run in GitHub's cloud and cannot access your Kubernetes cluster's internal services. Self-hosted runners (via ARC) run as pods inside your cluster, giving them direct network access to Ilum's internal API (ilum-core.ilum.svc.cluster.local).

Can I use this approach with GitLab CI or other CI/CD systems?

Yes. The core concept — calling the Ilum REST API from a CI/CD pipeline — works with any CI/CD system. You would need to use a GitLab Runner on Kubernetes (or similar) and adapt the workflow syntax to your CI/CD platform.

How do I scale the number of runners?

Increase the النسخ المتماثله field in runner-deployment.yaml. ARC also supports HorizontalRunnerAutoscaler for auto-scaling based on workflow queue depth.

Last updated on