Pebble vs LevelDB During an Avalanche C-Chain Replay

A fresh Avalanche C-Chain replay using Pebble sustained about 179 blocks per second over a recent 30-minute window. The previous LevelDB run averaged about 8.9 blocks per second and had fallen to roughly 4 to 5 blocks per second before it was stopped.

That is a 20 to 40 times difference in the observed windows. It is not a controlled benchmark. Pebble was processing the first 5.6 million blocks, while LevelDB had reached a later and more expensive part of chain history. The useful result is narrower: the current Pebble run executes blocks with a much better I/O profile, and neither run came close to saturating the NVMe device.

The observed comparison

Metric	Previous LevelDB run	Current Pebble run
C-Chain execution rate	8.9 blocks/s average; 4–5 blocks/s before shutdown	179 blocks/s over the latest 30 minutes
Apparent throughput difference	Baseline	About 20–40x faster
Disk writes	About 123 MB/s	About 318 MB/s over 5 minutes; peaks near 418 MB/s
Disk reads	About 89 MB/s	About 0.5–1.2 MB/s
Single-device utilization	About 8%–17%	About 2.3%–3.3%
System I/O wait	About 0.71%	About 0.05%
Data directory size	9.8 TB	662 GB at the current replay height
ETA reported in logs	About 1,428 hours	About 142 hours

The throughput, I/O rates, utilization, and I/O wait values are measurements from different replay periods. The directory size and ETA rows are operational context, not like-for-like efficiency measurements.

LevelDB and Pebble solve the same storage problem differently

Coreth, the EVM implementation behind Avalanche C-Chain, stores EVM state through AvalancheGo’s database interface and supports both PebbleDB and LevelDB backends ^[1]. AvalancheGo exposes the selection through its database type configuration ^[2].

Both engines are embedded, ordered key-value stores built around log-structured merge trees. LevelDB buffers writes in memory, records them in a log, flushes sorted tables to disk, and compacts those tables across levels ^[3]. Pebble follows the same broad model and was derived from the Go implementation of LevelDB, with additional design work inspired by RocksDB ^[4].

The differences are in the implementation details that matter under sustained database load. Pebble documents a more concurrent commit pipeline, L0 sublevels, flush splitting, and metadata structures intended to reduce read amplification or coordination costs in large LSM trees ^[4]. Those features make the observed result plausible, but they do not prove which mechanism produced the speedup on this node.

The NVMe device was not the limiting resource

The strongest result is not the blocks-per-second ratio. It is the relationship between throughput and device utilization.

Pebble wrote about 318 MB/s over the five-minute sample, more than twice the LevelDB write rate. Despite that, its device utilization stayed near 2.3% to 3.3%, and system I/O wait remained near 0.05%. LevelDB wrote less data per second, read about 89 MB/s, and kept the device busy for as much as 17% of the sample.

Higher write throughput with lower device utilization is consistent with larger or better-coalesced writes, fewer blocking reads, and less time spent waiting on scattered storage operations. The very low Pebble read rate also suggests that the current working set is being served efficiently from memory or from data already available to the process.

These are interpretations, not direct measurements of cache behavior. Confirming them would require database-level cache, compaction, stall, and table-read metrics together with device IOPS and latency distributions.

The LevelDB run also failed to saturate the drive. Even while LevelDB executed blocks and the Pebble container fetched historical blocks in parallel, the highest observed device utilization was about 17%. Replacing the NVMe device with a faster one would not address the dominant delay shown by these samples.

Database age changes the cost of replay

The old LevelDB directory had grown to 9.8 TB. A large late-stage archival database can contain many more sorted tables and a much larger state keyspace than a 662 GB database near the beginning of replay. More contract activity, state reads, trie updates, and compaction work can make later blocks materially more expensive to execute.

This supports two possible explanations for the old run:

LevelDB was performing more random or amplified reads as the database and state history grew.
Later C-Chain blocks required more execution and state work independent of the storage engine.

The available host metrics cannot separate those effects. Treating the 9.8 TB and 662 GB directory sizes as a storage-efficiency comparison would also be incorrect because the databases represent different replay heights.

Avalanche documentation warns that historical data does not carry over when changing database types. A switch from LevelDB to PebbleDB requires a fresh database and replay rather than an in-place conversion ^[5]. That operational boundary is why the Pebble run currently covers earlier history.

Why the 20–40x result will narrow

Three uncontrolled variables favor the current Pebble sample:

Pebble is executing early blocks, while LevelDB was processing later blocks with more transactions, contracts, and accumulated state.
The Pebble database is much smaller and has not yet reached the table count or compaction pressure of the old archive.
The LevelDB storage sample included concurrent block fetching by the Pebble container.

The log ETA has the same limitation. An estimate of 142 hours instead of 1,428 hours reflects the current measured rate and current position. It is not a reliable completion forecast if execution cost rises with block height.

Pebble should therefore be expected to slow as replay advances. The present data does not establish a permanent 20 to 40 times advantage, nor does it isolate the storage engine from chain-era effects.

The operational decision

Continuing the Pebble replay is the right decision for this node. The current engine is processing blocks much faster, producing far fewer reads, and spending less time waiting on storage even while writing more data. The evidence also rules out NVMe bandwidth saturation as the reason the LevelDB replay was slow.

The next useful comparison should use matched block ranges. For each engine, record:

starting and ending C-Chain height;
blocks per second over the same interval length;
transaction count and gas used for the interval;
database size and sorted-table count;
cache hits, compaction bytes, write stalls, and read amplification;
device read/write IOPS, latency, throughput, and utilization;
CPU time, memory pressure, and system I/O wait;
every other process using the same device.

Until that matched test exists, 179 blocks per second is an encouraging early-run result rather than a forecast. Pebble has earned the right to continue; its long-term advantage still needs to be measured at the same part of chain history that slowed the LevelDB database.