Manifest Bloat in Apache Iceberg: How to Detect It and When to Run rewrite_manifests
Iceberg manifest files are the index your query engines read before touching a single data file. When manifests bloat (thousands of small manifest files per snapshot), query planning slows significantly before any data is read. Here is what causes manifest bloat and when rewrite_manifests actually helps.
What Iceberg manifests are and why they matter
Every Iceberg snapshot has a manifest list: a file that points to one or more manifest files. Each manifest file lists the data files for a subset of the table’s partitions, along with statistics (min/max values, null counts) that query planners use for partition pruning. The deeper the manifest tree, the more files a query engine must read before it can begin scanning data.
How manifest bloat develops
Manifest bloat happens when writes are small and frequent. Every write appends a new manifest to the snapshot’s manifest list, rather than merging into existing manifests. After thousands of incremental writes without a rewrite_manifests pass, a single table scan requires reading hundreds of manifest files before reaching any data, adding seconds to query startup time.
- Streaming pipelines with micro-batch commits: each commit adds a new manifest entry
- Frequent small appends without compaction: each append creates a new manifest rather than updating existing ones
- No rewrite_manifests in the maintenance schedule: manifests accumulate until query planning visibly degrades
When to run rewrite_manifests
Run rewrite_manifests when query planning time has increased without a corresponding increase in data volume. rewrite_manifests compacts the manifest files and updates the snapshot pointer. The operation is safe to run while the table is being read.