Apache Iceberg

Apache Iceberg is an open-source, high-performance table format designed for massive analytic datasets. Originally developed by Netflix to address the performance and usability limitations of traditional Hive table formats, Iceberg was later donated to the Apache Software Foundation. It brings the reliability and SQL-like behavior of traditional relational databases to object storage systems like Amazon S3, Azure Blob Storage, and Google Cloud Storage, as well as distributed file systems like HDFS.

Historically, data lakes relied on the Apache Hive table format, which defines a table as a directory of files. While this approach worked for early big data architectures, it introduced significant bottlenecks at scale. Hive tables require expensive directory listing operations to identify which files belong to a table, a process that degrades in performance as the number of files grows. Furthermore, Hive lacks atomic transaction support, meaning readers can see partial or inconsistent data if a write operation fails mid-way. Schema changes in Hive can also lead to silent data corruption if not managed with extreme care.

Apache Iceberg solves these challenges by tracking data at the individual file level rather than the directory level. By maintaining a highly structured, hierarchical metadata tree, Iceberg allows query engines to safely work with the same tables concurrently. It supports full ACID (Atomicity, Consistency, Isolation, Durability) transactions, ensuring that writes are atomic and readers always see a consistent snapshot of the data. Additionally, Iceberg abstracts the physical layout of the data from the user, enabling advanced features like hidden partitioning, schema evolution, and time travel without requiring complex data migrations or manual query adjustments.

The architecture of Apache Iceberg is structured as a hierarchical tree of metadata and data files. This design decouples the logical view of the table from the physical files stored on disk. The architecture is divided into three distinct layers: the Catalog, the Metadata Layer, and the Data Layer.

At the top of the hierarchy sits the Iceberg Catalog. The catalog's primary responsibility is to maintain state by tracking the current metadata pointer for a given table. When a query engine wants to read or write to an Iceberg table, it first queries the catalog to locate the current table metadata file. Because the catalog must guarantee atomic state transitions, it is typically backed by a transactional system such as a relational database (via JDBC), the Hive Metastore, AWS Glue, Project Nessie, or a REST-based catalog service.

Below the catalog is the Metadata Layer, which consists of three types of files:

1. Table Metadata Files: These files store the table's schema, partition specifications, and a history of snapshots. Each write operation creates a new table metadata file, and the catalog is updated to point to this new file. This file acts as the root of a specific table version.

1. Manifest Lists: A manifest list defines a single snapshot of the table. It contains an entry for every manifest file that makes up that snapshot. Crucially, the manifest list stores partition-level summary statistics for each manifest file. This allows query engines to prune entire manifest files during query planning without reading them from storage.

1. Manifest Files: Manifest files track individual data files. Each manifest file records metadata about a set of data files, including their physical paths, row counts, and column-level statistics (such as minimum and maximum values, null counts, and nanosecond-level timestamps). These statistics enable fine-grained file pruning, ensuring that the query engine only reads the specific data files containing relevant data.

At the bottom is the Data Layer, which contains the actual data files. Iceberg is format-agnostic at the data layer, supporting popular columnar formats like Apache Parquet and Apache ORC, as well as row-based formats like Apache Avro. This layer also stores delete files, which track deleted or updated rows when using Merge-on-Read configurations.

When a write operation occurs, Iceberg writes new data files, generates new manifest files, creates a new manifest list, and writes a new table metadata file. Finally, it attempts to commit the transaction by updating the catalog pointer. If another write committed during this process, Iceberg uses optimistic concurrency control to retry the commit automatically, merging the changes if no conflicts exist.

Apache Iceberg offers several significant advantages that make it a compelling choice for modern data platform architectures:

• ACID Transactions and Concurrency: Iceberg supports multi-table transactions and guarantees snapshot isolation. Readers always query a consistent snapshot of the table, unaffected by concurrent write operations. Optimistic concurrency control ensures that concurrent writers can commit safely, with automatic conflict resolution.

• Full Schema Evolution: Unlike traditional formats where renaming or dropping columns can corrupt historical data, Iceberg supports safe, in-place schema evolution. Columns can be added, dropped, renamed, reordered, or have their types promoted without rewriting the underlying data files. Iceberg achieves this by assigning unique, immutable IDs to each column, mapping them independently of physical names or positions.

• Partition Evolution: As data volume grows or query patterns change, the optimal partitioning strategy for a table may change. Iceberg allows data platform engineers to update the partition layout of a table over time without rewriting historical data. Queries automatically handle the transition, applying the old partition layout to old data and the new layout to new data.

• Hidden Partitioning: In traditional systems, users must know how a table is partitioned and write queries that explicitly filter on partition columns to avoid full table scans. Iceberg introduces hidden partitioning, where the relationship between the source column and the partition value is stored in metadata. Users query the source column directly, and Iceberg automatically applies partition pruning.

• Time Travel and Rollback: Because Iceberg retains historical snapshots, users can query the state of a table at a specific point in time or snapshot ID. This is invaluable for auditing, reproducing machine learning models, and recovering from accidental data corruptions.

• Engine Agnostic: Iceberg is designed as an open standard and is not tied to a single processing engine. It features robust, official integrations with Apache Spark, Apache Flink, Trino, Presto, Hive, and various commercial data platforms, allowing different engines to safely read and write to the same tables simultaneously.

While Apache Iceberg provides powerful features, it introduces specific trade-offs and operational complexities that organizations must manage:

• Write Amplification and Delete Overhead: Iceberg supports two modes for handling updates and deletes: Copy-on-Write (CoW) and Merge-on-Read (MoR). CoW rewrites entire data files even if only a single row changes, leading to high write amplification. MoR writes deletes to separate files, reducing write amplification but increasing read amplification, as the query engine must merge data and delete files on the fly. Managing this balance requires careful tuning.

• Operational Maintenance: Iceberg tables accumulate historical snapshots, metadata files, and delete files over time. Without regular maintenance, storage costs can escalate, and query planning performance can degrade. Teams must set up automated maintenance pipelines to regularly expire old snapshots, clean up orphan files, and compact small data files.

• Catalog Dependency: Iceberg relies heavily on its catalog to maintain transactional integrity. If the catalog becomes unavailable, all read and write operations on the tables cease. Furthermore, migrating between different catalog types (e.g., from Hive Metastore to AWS Glue) requires careful planning and execution.

• Small File Problem: Frequent streaming writes or small batch ingestions can result in thousands of tiny files. While Iceberg handles metadata efficiently, a massive number of small files still degrades query performance. Users must run compaction jobs (such as Spark's rewriteDataFiles action) to merge small files into larger, optimal-sized files.

• Learning Curve and Tooling Maturity: Transitioning from traditional directory-based tables to file-level metadata tables requires a shift in mindset for data engineers. While major engines support Iceberg, smaller or older tools in the data ecosystem may lack native integration, requiring custom connectors or intermediate views.

Apache Iceberg is highly suited for modern data platform architectures that demand reliability, flexibility, and multi-engine compatibility:

• Multi-Engine Data Lakehouses: Organizations that use different engines for different workloads benefit immensely from Iceberg. For example, a team can use Apache Flink for real-time streaming ingestion, Apache Spark for heavy batch processing, and Trino or Presto for interactive SQL queries, all operating on the same Iceberg tables without risk of data corruption.

• Regulatory Compliance and Data Privacy: Under regulations like GDPR and CCPA, organizations must support the "right to be forgotten" by deleting specific user records. Iceberg's support for row-level deletes (via Merge-on-Read or Copy-on-Write) allows data engineers to target and delete specific records efficiently without needing to rebuild entire data pipelines or rewrite massive datasets.

• Reproducible Machine Learning and Auditing: Data science workflows often require training models on exact historical snapshots of data to ensure reproducibility. Iceberg's time-travel capability allows data scientists to query a table exactly as it existed at a specific timestamp, ensuring consistent model training inputs.

• High-Throughput Streaming Ingestion: Iceberg's commit protocol allows continuous streaming ingestion from sources like Kafka or Flink. Because readers are isolated from writers, analytical queries can run continuously against the table without experiencing lock contention or reading partial, uncommitted data.

Despite its strengths, Apache Iceberg is not a universal solution and should be avoided in the following scenarios:

• Low-Latency OLTP Workloads: Iceberg is designed for analytical processing (OLAP) on massive datasets. It is not a replacement for transactional relational databases (like PostgreSQL or MySQL) or NoSQL databases (like Cassandra or DynamoDB). If your application requires sub-second point lookups, single-row inserts, or high-frequency transactional updates, Iceberg is inappropriate.

• Small Datasets: If your total data volume is small (e.g., under a few hundred gigabytes) and fits easily within a single relational database or can be managed with simple Parquet files on a local filesystem, the architectural overhead of maintaining an Iceberg catalog and running regular maintenance jobs outweighs the benefits.

• Environments Without Centralized Infrastructure: Iceberg requires a reliable catalog service to manage state. If your deployment environment is highly decentralized, lacks a persistent catalog, or cannot run background maintenance tasks (like compaction and snapshot expiration), managing Iceberg tables will become operationally challenging.

• Legacy Toolchains: If your organization relies heavily on legacy BI tools or proprietary data warehouses that do not natively support Iceberg and cannot easily integrate with query engines like Trino or Spark, adopting Iceberg may introduce integration friction that offsets its performance advantages.