تخطي إلى المحتوى الرئيسي

DuckLake

DuckLake is the main storage solution for the DuckDB database engine used by Ilum.

Since embedded DuckDB by default supports only in-memory mode or local disk storage (single user), DuckLake provides a multi-user, concurrent storage layer that enables shared access to DuckDB datasets across the platform.

When to use DuckLake​

Use DuckLake when:

  • Multiple users or jobs need concurrent access to the same DuckDB tables
  • You need persistent table storage with metadata management
  • Your workload requires time travel or schema evolution

Skip DuckLake when:

  • You have single-user, ad-hoc analytics workflows
  • You’re prototyping and don’t need persistence
  • Direct Parquet file access is enough

ملامح​

  • Shared dataset storage using object storage (S3/MinIO/GCS)
  • ACID-like durability through snapshot isolation and cross-table transactions
  • Time travel – query previous versions of your data
  • تطور المخطط – add/modify columns without breaking existing queries
  • التوافق with standard DuckDB file formats and Parquet

For more on DuckLake’s capabilities, see the official DuckLake documentation.

Concurrency Model​

DuckLake provides snapshot isolation for concurrent workloads:

  • Readers never block writers, and writers never block readers
  • Each query sees a consistent snapshot of data at transaction start
  • Cross-table transactions maintain atomicity across related operations

This differs from strict serializable ACID – it’s optimized for analytic workloads where high read concurrency is prioritized over write serialization.

تكوين​

ملاحظه

DuckLake is attached by default to all DuckDB instances in Ilum when enabled. Tables created in SQL Viewer will automatically use DuckLake.

Configure DuckLake via the Helm chart:

إيلوم كور:
  SQL:
    duckdb:
      ducklake:
        تمكين: صحيح                          # Set false to disable DuckLake entirely
        location: ق3://ilum-ducklake/          # Root path for all DuckLake data
        Postgres:                              # Metadata storage (required)
          مضيف: "ilum-postgresql-hl"           # PostgreSQL service hostname
          ميناء: 5432                           # PostgreSQL port
          database: ducklake                   # Database name (created automatically if missing)
          مستخدم: إيلوم                           # Database user
          شعار: "CHANGEMEPLEASE"           # Database password
        ق3:                                   # Data storage backend
          نقطه النهايه: إيلوم-مينيو:9000           # S3 endpoint (MinIO, AWS S3, GCS, etc.)
          region: us-شرق-1                   # منطقة S3
          keyId: minioadmin                   # Access key ID
          سر: minioadmin                  # Secret access key
          urlStyle: مسار                      # Path-style access (use 'virtualHost' for AWS S3)
          ssl: خطأ                          # Enable TLS for S3 connections
Disabling DuckLake

Setting enabled: false causes DuckDB to fall back to in-memory tables only. Use this only for single-user scenarios where persistence isn’t required.

أمثلة الاستخدام​

Creating Tables​

When enabled, DuckLake is automatically selected as the default storage backend for all DuckDB instances. This means any created tables in the default catalog will be stored in DuckLake.

-- Create a table (automatically stored in DuckLake)
خلق جدول events (
    event_id بيجينت,
    event_time الطابع الزمني,
    user_id بيجينت,
    payload JSON
);

Limitations​

  • DuckDB-only: DuckLake catalogs cannot be accessed from Spark, Trino, or other engines
  • No branching/tagging: Unlike Nessie or Iceberg, DuckLake doesn't support Git-like version control
  • Concurrency bound by metadata DB: High-write concurrency may impact PostgreSQL performance
Last updated on