كتالوج نيسي

نظره عامه ​

Project Nessie هو مفتوح المصدر كتالوج بيانات المعاملات that brings Git-like version control to data lakes. It enables you to manage multiple versions of your data using branches, tags, and commits—similar to how Git manages source code.

With Ilum’s integration, you can leverage Nessie’s version control features directly in your Spark environment. This allows you to branch, tag, and merge data changes safely and efficiently.

Unlike traditional Hive or Glue catalogs, which only track the latest state of each table, Nessie records جميع التغييرات حسب الالتزام in a timeline. Each commit represents a consistent snapshot of your data lake. Changes are isolated until committed, ensuring incomplete or in-progress updates are never visible to other users or jobs. Once finalized, changes become atomically visible, guaranteeing consistency.

Key Features: Nessie vs. Traditional Catalogs​

Highlights​

• Branching: Create multiple isolated branches (e.g., رئيسي , ديف , staging) without duplicating data. Branches are lightweight pointers to metadata snapshots.
• البيئات المعزولة: Use the same data lake for dev, staging, and prod by isolating changes in branches. No need for separate catalogs or data copies.
• الالتزام بالتاريخ والسفر عبر الزمن: Nessie maintains a unified commit log. Inspect, audit, or time-travel to any previous state by commit hash or timestamp.
• المعاملات الذرية متعددة الجداول: Commit changes across multiple tables as a single atomic operation. All succeed or none do.
• Collaboration & Governance: Work on separate branches, merge changes, and track who changed what and when. Enables safe experimentation and robust auditability.

Core Concepts​

فروع ​

A فرع is an independent line of development for your data catalog. Branches start as copies of existing branches and track changes separately. They are lightweight, referencing the same data files but different metadata. The default branch is usually رئيسي .

العلامات ​

A العلامه is a read-only label pointing to a specific commit. Use tags to mark stable versions or important milestones (e.g., الإصدار 1.0 , إصدار 2025-06 ). Tags are immutable bookmarks.

Commits​

A ارتكب is a set of changes recorded as a single atomic unit. Each commit has a unique ID, timestamp, author, and optional message. The commit log provides full catalog versioning.

استخدام Nessie في Ilum ​

يدعم Ilum مشروع نيسي as a catalog for version-controlled data management. When using Ilum notebooks or Spark jobs with Apache Iceberg, you can perform Git-like operations—branching, merging, tagging—directly via SQL.

Nessie requires special configuration with Spark. If used as the main Ilum catalog, the nessie_catalog should be pre-configured in your Spark session.

Additionally, make sure the Spark image used in your cluster has the Nessie client installed. In particular, you will need:

• Iceberg Spark Runtime (org.apache.iceberg:iceberg-spark-runtime-<SPARK_VERSION>_<SCALA_VERSION>) - Required for Nessie support.
• Iceberg AWS Bundle (org.apache.iceberg:iceberg-aws-bundle) - Required for S3 support.
• Nessie SQL Extensions (org.projectnessie.nessie-integrations:nessie-spark-extensions-<SPARK_VERSION>_<SCALA_VERSION>) - Required for Nessie-specific SQL operations.

You can also use Ilum’s custom Spark image for Nessie: ilum/spark:<SPARK_VERION>-nessie, which includes all the required dependencies.

Additionally, in order for Nessie’s AWS S3 support to work, you need to pass the credentials to the Spark instance differently than with the Hive&Delta solution. To pass the credentials, you can use the spark.driver.extraJavaOptionsو spark.executor.extraJavaOptions Spark configuration options. In particular, the following options should be appended:

• -Daws.region=<REGION> - The default Minio region is الولايات المتحدة الشرقية 1 .
• -Daws.accessKeyId=<ACCESS_KEY_ID> - The default Minio access key is minioadmin .
• -Daws.secretAccessKey=<SECRET_ACCESS_KEY> - The default Minio secret key is minioadmin . For a more secure solution, the credentials should be built into the Spark image.

Nessie Walkthrough​

In the beginning, it is recommended to create anything inside the main branch so that you avoid problems with merging into an empty branch:

خلق  جدول nessie_catalog . المستعملون ( 
    user_id الباحث , 
    user_name فارشار ( 20 )   
) ; 

Create a Branch​

خلق تطوير الفرع في nessie_catalog من رئيسي ; 

And to verify everything, list all branches and tags:

قائمة مراجع  في nessie_catalog ; 

Work on a Branch​

Create a table and insert data in the ديف branch with the fully qualified name (<TableName>@<BranchName>أو <TableName>@<CommitHash>):

خلق  جدول nessie_catalog . ` sales@dev ` ( 
    sale_timestamp CHAR( 10 ) , 
    sale_amount الباحث , 
    payment_method فارشار ( 20 ) 
) ; 

أدخل  إلى nessie_catalog . ` sales@dev `  القيم 
    ( '2025-06-01' ,  1000 ,  "عبر الإنترنت" ) , 
    ( '2025-06-02',  1500,  'InStore') , 
    ( '2025-06-03',  800,  "عبر الإنترنت" ) , 
    ( '2025-06-04',  1200,  'Mobile') , 
    ( '2025-06-05',  950,  'InStore') ; 
اختار  عد ( * )  من nessie_catalog . ` sales@dev ` ; 

Or use the استخدام statement to switch a context to a specific branch:

استخدام تطوير الفرع في nessie_catalog ; 
اختار  عد ( * )  من nessie_catalog . مبيعات ; 

And show the log of all commits done:

عرض  LOG على ديف في nessie_catalog ; 

Merge Branches​

دمج تطوير الفرع إلى رئيسي في nessie_catalog ; 
عرض  المناضد  في nessie_catalog ; 

أفضل الممارسات ​

• Develop in Isolation: Use branches for development or experiments. Promote changes through a hierarchy (e.g., dev → staging → main).
• Merge Frequently: Merge changes regularly to minimize conflicts.
• حافظ على الفروع قصيرة الأجل: Remove feature branches after merging.
• Avoid Conflicts: Sync your branch with the latest target branch before merging.
• Tag Milestones: Use tags for stable releases or important checkpoints.
• Document Changes: Add commit messages for traceability.

التعرف على المزيد ​

For advanced SQL operations and the full Nessie Spark SQL reference, see:
👉 مرجع Nessie Spark SQL

Nessie with Ilum combines Spark’s power with Git-like data management, enabling robust “data as code” workflows for your lakehouse.