تخطي إلى المحتوى الرئيسي

Exporting Notebooks to Pipelines · JPE

ال JupyterLab Pipeline Exporter (JPE) is a JupyterLab 4 extension that promotes an interactive Sparkmagic notebook into a production artifact - an Airflow DAG, or a native Ilum Spark job in single, service, or cron mode - without leaving the notebook UI. The notebook is parsed in place, Jupyter-only constructs are stripped, and the resulting code is shipped to the configured backend.

JPE ships pre-installed in the helm_jupyter sub-chart. On a standard Ilum deployment, every backend URL is auto-wired from in-cluster Service DNS; no manual configuration is required.

Deployment Targets​

JPE offers two export targets, each with its own execution modes. Both target cards are enabled when their backend responds at panel load; the backend is probed and reflected as a status badge.

TargetModesMechanismBacking API
Ilum Spark jobsingle · service · cronNative Ilum. The selected mode determines how the notebook is packaged and submitted to إيلوم كور.POST /api/v1/job/submit, POST /api/v1/group, POST /api/v1/schedule
Airflow DAGper-cell · batchRenders a DAG Python file via Jinja2 using the Livy operator, pushes it to Gitea, and lets git-sync deliver it to the Airflow dag-processor. Optional auto-trigger after visibility.Gitea Contents API + Airflow REST API (/api/v2/dags)

Three further targets - SDP (Spark Declarative Pipelines), dbt projectو DuckDB via Quack - are visible in the panel as coming soon and are not yet selectable.

Independently of the target, any notebook can be downloaded as a Spark-runnable .py ملف for inspection or manual شرارة تقديم; this download is always available, including fully offline.

Execution Modes​

Ilum Spark job​

ال Ilum Spark job target submits to إيلوم كور in one of three modes, chosen with the mode toggle on the target card:

  • single - Wraps the notebook as a standalone pyFile and submits it as a one-shot job on an ephemeral Spark application that is stopped automatically when the job completes. (POST /api/v1/job/submit)
  • خدمة - Wraps the notebook as an Ilum Job subclass and registers a long-running Ilum service with a warm Spark driver, then runs the notebook against it on demand. (POST /api/v1/group, executed via POST /api/v1/group//job/execute)
  • cron - Registers the service together with a cron expression. Each fire is a single-shot Spark job triggered by إيلوم كور's internal scheduler - no Kubernetes CronJob is created. (POST /api/v1/schedule)

Airflow DAG​

ال Airflow DAG target renders in one of two shapes:

  • per_cell - Default. Each code cell becomes one PythonOperator task chained sequentially; all tasks share a single Livy session opened by a leading livy_session task and closed by a trailing livy_cleanup task. Fine-grained DAG with individually retriable cells. More Livy round-trips than batch.
  • batch - A single PythonOperator task (run_notebook) runs the whole notebook inside one Livy session as one Spark batch job. Minimizes session startup overhead at the cost of cell-level retriability.

Cell Tags​

Two aspects of the generated Airflow DAG are controlled per cell through JupyterLab cell tags. Tags are read from the notebook metadata; they are inert during interactive execution and only consumed at export time.

TagApplies toEffect
task:per_cell task idReplaces the generated id (cell_3أو c2_c3 for merged cells) with a stable, readable task_id.
retries:Per-task retry countOverrides the DAG-level default retry count for that task.

Naming per-cell tasks​

في per_cell mode each task defaults to a generated id. Add a task: cell tag (for example task:load_raw_sales) to assign a stable task_id. The name must be a valid Python identifier; invalid tags are ignored and fall back to the generated id. The active panel surfaces a Name your tasks hint while per_cell is selected, and the export preview shows a tagged-cell counter.

Per-task retry count​

Generated DAGs carry a DAG-level default of three retries with exponential backoff. To override the retry count for an individual task, add a retries: cell tag, where is a non-negative integer (for example retries:5). A flaky ingestion step can be given more retries while a deterministic transform is set to retries:0 to fail fast.

The value is applied per task:

  • في per_cell mode the tag sets retries= on that cell's PythonOperator, overriding the DAG default for that task only.
  • في batch mode there is a single run_notebook task; the first valid retries: tag among the exported cells applies to it.

A tag whose value is not a non-negative integer (for example retries:-1, retries:abc, retries:2.5) is ignored, and the task keeps the DAG-level default. Surrounding whitespace is tolerated (retries: 3). When multiple retries: tags are present on merged cells, the first valid one wins.

# Cell tags: task:load_raw_sales, retries:5
raw = شراره.قرأ.الباركيه("s3a://landing/sales/")
raw.يكتب.saveAsTable("bronze.sales")

The cell above renders as:

load_raw_sales = PythonOperator(
    task_id="load_raw_sales",
    python_callable=_submit_statement,
    op_kwargs={"livy_conn_id": "ilum-livy-proxy", "code": "..."},
    retries=5,
)

The shipped example notebooks (Pipeline_Exporter_Showcase, Pipeline_Exporter_Ilum_Service_Showcase, and the docker-compose quickstarts) demonstrate both tags, assigning higher retry counts to I/O-bound steps and retries:0 to parameter cells.

Operating Modes​

Each backend is probed when the panel loads (≤1.5 s) and reflected as a status badge on the target card.

  • Standalone mode - No remote backend responds. Both target cards are dashed-bordered with a tooltip, and only the .py download is offered. Typical for a pip install jupyterlab-pipeline-exporter setup whose backend URLs have not been pointed at an Ilum or Airflow instance yet.
  • Connected mode - Some backends are reachable. Cards light up independently.
  • Ilum bundled mode - All backends auto-discovered: ILUM النواة: 9888, ilum-airflow-api-server:8080, ilum-gitea-http:3000. DAG auto-trigger mints HS512 tokens locally from AIRFLOW_JWT_SECRET (the FAB+OAuth /auth/token endpoint is incompatible with Airflow 3.x).

Notebook Sanitization​

A Sparkmagic notebook contains constructs that crash an Airflow worker - no IPython display backend, no ٪ السحر resolver, no shell. JPE strips these before generating any artifact and reports every stripped line with its cell index and line number.

Removed by default:

  • Line magics - النسبة المئوية manage_spark, %matplotlib, %load_ext
  • Shell escapes - !pip install
  • Display calls - print(...), display(...) with no terminal sink in batch execution
  • Secret-looking literals - AWS keys, JWT payloads, common password patterns. The request is rejected with HTTP 400 unless allow_secrets: true is set.

Stripped lines appear in the rejected list of every response and in the preview panel. Set keep_rejections: true to retain a magic or display call (for example a طبع that writes to a log aggregator).

بقشيش

استخدام Preview before committing. It returns the fully rendered DAG or pyFile together with the rejection report - without pushing to Gitea or Ilum.

Cell-kind filtering and the exported-cell count​

ال enabledCellKinds setting decides which cell kinds (شراره, بايسبارك, SQL, سكالا, plain) are exported. The JupyterLab panel defaults to ["spark", "pyspark"], so a Sparkmagic notebook exports only its ٪٪شرارة / %%pyspark cells; plain Python and ٪٪sql cells are dropped and listed in rejected مثل filtered-cell:. A parameter cell is always retained regardless of the filter.

A cell that passes the kind filter but sanitizes to an empty body - for example a setup cell containing only line magics such as ٪ load_ext sparkmagic.magics و النسبة المئوية manage_spark - produces no task. It is reported in rejected مثل emptied-cell and excluded from the count.

Because of this, the cells value returned by an export is the number of cells that actually became tasks (or, for batch and pyFile targets, statements) in the generated artifact - not the raw count of code cells in the notebook. Skipped cells, kind-filtered cells, and emptied setup cells do not contribute to it. If every exportable cell is filtered out, the request is rejected with HTTP 422 so an empty pipeline is never produced.

Pipeline Parameters​

JPE recognizes parameter cells through two conventions:

  • A cell tagged parameters (the papermill convention)
  • A cell whose first non-blank line is # @pipeline-params (kebab) or # @pipeline_params (snake). The marker is a plain Python comment, so the cell executes interactively in any IPython kernel without raising UsageError: Cell magic … not found

Variables assigned in such a cell are emitted as config.get('', ) lookups in the generated code, so the same notebook runs unchanged interactively (literal default) and in production (values injected by Airflow or the Ilum service request).

# Tagged: parameters
output_table = "gold.daily_kpi"
run_date = "2026-05-21"
threshold = 0.95

In the generated artifact the same variables resolve to:

output_table = التكوين.حصل("output_table", "gold.daily_kpi")
run_date = التكوين.حصل("run_date", "2026-05-21")
threshold = التكوين.حصل("threshold", 0.95)

History and Re-Execution​

Every submission is recorded in the History tab with the export specification, generated code, and rejection report. The same record is persisted to ~/.jupyter/jpe-audit.jsonl for audit.

For Ilum service submissions JPE also keeps a re-execution payload - the service can be re-invoked from the History tab without re-uploading the pyFile or recreating the service.

تكوين​

All settings are in Settings → Plugin Settings → Pipeline Exporter. Defaults target in-cluster Ilum services; everything is overridable.

Settingافتراضيوصف
ilumApiUrlhttp://ilum-core:9888/api/v1Ilum-core REST API used by single, service, and cron targets.
airflowApiUrlhttp://ilum-airflow-api-server:8080Airflow REST API used for DAG visibility checks and auto-trigger.
gitApiUrlhttp://ilum-gitea-http:3000/api/v1Git provider API used to push generated DAGs.
gitProviderجيتاGit provider type - currently Gitea in the bundled deployment.
defaultClusterIdافتراضيIlum cluster name pre-selected in the dialog.
defaultLivyConnIdilum-livy-proxyAirflow connection ID used by generated operators.
defaultModeper_cellAirflow export shape pre-selected in the dialog (per_cell أو batch).
sparkImages6 presetsSpark image dropdown - Cluster default (empty, inherits the cluster's image) plus Spark 4.1.2 with Delta, Sedona, Iceberg, or Trino, and Spark 3.5.8 with Nessie + Sedona.
enabledCellKinds["spark", "pyspark"]Cell kinds (شراره, بايسبارك, SQL, سكالا, plain) included in the export. Cells outside this list are dropped and reported.

Override precedence (highest wins):

  1. Request payload fields - per submission.
  2. Pod environment variables (ILUM_API_URL, AIRFLOW_API_URL, GITEA_API_URL, GITEA_TOKEN, …).
  3. Plugin settings (JupyterLab Settings editor).
ملاحظه

On a standard Ilum deployment the Helm chart wires every URL, secret, and JWT into the Jupyter pod from helm_jupyter/values.yaml. The Settings editor is only needed for non-bundled deployments or per-user overrides.

  • airflowIntegration.enabled (default صحيح) gates the AIRFLOW_JWT_SECRET / AIRFLOW_API_URL env block. Set it خطأ on deployments without Airflow so the disabled state is explicit at template render time rather than a silently empty token at runtime.
  • git.existingSecret alone gates the GITEA_USERNAME / GITEA_PASSWORD env vars JPE uses to push DAGs. The init-container that seeds the work dir into Gitea is gated separately by git.initialCommit.enabled, so a slim deploy without the gitea sub-chart can keep JPE's Gitea credentials without forcing the init loop to run. git.enabled is retained as a deprecated alias for git.initialCommit.enabled.

بداية سريعة​

  1. Open the notebook in JupyterLab.
  2. Click the Pipeline Exporter icon in the left sidebar (rank 102, below the file browser).
  3. Pick the Ilum Spark job target card. Select خدمة mode.
  4. Enter a service name (for example daily-kpi-svc) and an optional parameter list.
  5. نقر Create Ilum Service. A green banner confirms that the service has been created.
  6. Open the History tab. The new service entry exposes a Run action.
  7. Observe the run under Workloads → Services في واجهة مستخدم Ilum.
# Generated wrapper (excerpt)
فصل NotebookService(Ilum Job):
    مواطنه ركض(ذات, شراره, التكوين):
        output_table = التكوين.حصل("output_table", "gold.daily_kpi")
        # ... notebook body, sanitized

Generated Artifacts​

Target · modeArtifactمكان
Airflow DAG.pyGitea repository (//:/) - picked up by git-sync and parsed by dag-processor.
Ilum Spark job · singlepipeline_exp_/main.pyUploaded to Ilum as part of the job submission; visible at /workloads/details/job/ في واجهة مستخدم Ilum.
Ilum Spark job · serviceNotebookService class in module pipeline_exp_Registered as the Ilum service's main class FQN; invoked via POST /api/v1/group//job/execute.
Ilum Spark job · cronService + schedule entrySame as service plus a row in /schedules; each fire is a single-shot job triggered by إيلوم كور.
.py download (any target)Wrapped .py ملفBrowser download - useful for inspection, manual شرارة تقديم, or offline development.
تحذير

Service and cron modes wrap the notebook as an Ilum Job subclass, whose base class comes from the ilum-python-job package (from ilum.api import IlumJob). The Spark image used to run the service must provide this package - the Ilum Spark runtime images ship it. If a custom image is used in service or cron mode, install ilum-python-job into it; otherwise the import fails at driver startup. The single mode and the Airflow DAG target do not require it.

Last updated on