Interactive Spark Code Service on Kubernetes

Step-by-Step Guide: Setting Up an Interactive Spark Session​

Here's a guide to launching an interactive Spark environment using the Ilum dashboard.

1. Navigate to Services: Access the 'Services' section in your Ilum dashboard.

2. Create New Service: Click the "New Service +" button to start setting up your interactive service environment.

3. Configure General Settings:

   • اسم : Enter a unique name for your service (e.g., MyCodeService)
   • عنقود : Select the Kubernetes cluster where you wish to run your interactive service.
   • نوع : Select رمز to initialize the interactive REPL environment.
   • اللغة : Choose سكالا أو بايثون (PySpark).

   

4. Optional Configuration: Navigate through the tabs to configure resources, Spark configurations, and dependencies. See the Interactive Service Configuration Reference below for details.

5. Activate the Service: Click خلق to launch the persistent Spark session.

6. Locate Your Service: Once the service is activated, find it in the Services list. The service may take a few moments to start up and will show a "Starting" status. Wait until the status shows "Running" before proceeding. Click on the service to view the service details.

Creating Interactive Spark Services via REST API​

You can create interactive Spark services programmatically via the Ilum API, enabling automation through scripts, CI/CD pipelines, or orchestration tools.

Create Code Service​

To create a Code type service, use the /api/v1/group endpoint with بيانات متعددة الأجزاء / النماذج :

curl -X POST 'http://localhost:9888/api/v1/group' \
  --form 'name="MyCodeService"' \
  --form 'type="code"' \
--form 'clusterName="default"' \ 
  --form 'language="PYTHON"' \
  --form 'scale="1"'

curl -X POST 'http://localhost:9888/api/v1/group' \
  --form 'name="MyScalaCodeService"' \
  --form 'type="code"' \
--form 'clusterName="default"' \ 
  --form 'language="SCALA"' \
  --form 'scale="1"'

Response:

{ 
  "معرف المجموعة" :  "20251222-1209-bw4pf"
} 

مهم: Save the معرف المجموعة value - you'll need it for code execution and status checks!

API Creation Fields​

Verify Service Creation​

After creating the service, verify it's running:

curl http://localhost:9888/api/v1/group

Wait for service حالة to be ACTIVE and job حالة to be READY before executing code. This typically takes 10-15 seconds after creation.

Using the Interactive Execution Panel​

1. Open the Execution Panel: Click on the "Execute Job" button in the service details view.

2. Write Code: Enter your Scala or Python code in the editor.

3. أعدم : Run the code to see immediate results.

   

   The execution panel allows you to run Spark code and view results in real-time.

4. Verify Results: Output is displayed below the editor.

   

Execute code fragments programmatically. Replace {{groupId}} with your service's group ID:

curl -X POST "http://ilum-core:9888/api/v1/group/{{groupId}}/job/execute" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "interactive_code_execute",
    "code": "2+2"
}' 

General Tab​

• اسم: A unique identifier for the interactive service. This name will be used to track the service's status and logs within the Ilum dashboard.
• Cluster: The specific cluster where the service's resources will be allocated. Ensure the selected cluster has sufficient capacity for your interactive session.
• وصف: A brief explanation of the service's purpose. This helps other users understand what the service is used for.
• Type: The operational mode of the service. Select رمز to establish an interactive REPL environment for running ad-hoc code snippets, or مهمة to run a specific job class interactively.
• Language: The programming language used for the service (سكالا أو بايثون ). This determines the runtime environment for your code.
• Scale: The number of replicas (instances) to launch for this service. Increasing the scale allows for load balancing and higher concurrency if multiple users are accessing the same service.
• Auto Pause: Automatically scales the service to zero replicas after a specified period of inactivity. This helps conserve resources when the service is not in use.

Configuration Tab​

• Parameters: Key-value pairs for configuring Spark properties (e.g., spark.sql. الامتدادات ). These settings override the default cluster configurations for this specific service.
• Tags: Custom labels to categorize and filter your services. Tags are useful for organizing resources by project, team, or environment (e.g., ديف , تحليلات ).

Resources Tab​

• Jars: Additional JAR files to be included in the classpath of the driver and executors. These libraries will be available for import in your interactive code.
• Files: Auxiliary files to be placed in the working directory of each executor. These can be configuration files or data files needed by your code.
• PyFiles: Python dependencies such as .zip, .eggأو .py files. These are added to the PYTHONPATH for Python-based interactive sessions.
• Requirements: A list of additional Python packages to install on the nodes. This ensures that all necessary libraries are available for your Python environment.
• Spark Packages: Maven coordinates for Spark JAR packages to be downloaded and included. This is a convenient way to include libraries from Maven Central without manually uploading JARs.

Memory Tab​

• Executors: The number of executor instances to allocate per replica. More executors allow for greater parallelism during data processing.
• Driver Cores: The number of CPU cores assigned to the driver process. This affects the driver's ability to handle task scheduling and result collection.
• Executor Cores: The number of CPU cores assigned to each executor. This determines how many tasks each executor can run simultaneously.
• Driver Memory: The amount of RAM allocated to the driver. Sufficient memory is crucial for the driver to manage application state and large results.
• Executor Memory: The amount of RAM allocated to each executor. This directly impacts the amount of data that can be cached and processed in memory.
• Dynamic Allocation: Enables the automatic scaling of the number of executors based on workload. When enabled, Ilum will request more executors when busy and release them when idle.
• Initial Executors: The initial number of executors to start with when dynamic allocation is enabled. This provides a baseline capacity when the service starts.
• Minimal number of executors: The lower bound for the number of executors when dynamic allocation is enabled. The service will never scale below this number.
• Maximal number of executors: The upper bound for the number of executors when dynamic allocation is enabled. This prevents the service from consuming too many cluster resources.