Storage Engine Internals

A storage engine is the component inside a database that actually writes data to and reads it from disk. The two dominant designs are B-tree-based engines, which update data in place and favor reads, and LSM-tree-based engines, which append writes and merge them in the background to favor high write throughput.

A B-tree-based engine keeps keys in a balanced, sorted tree and updates pages in place. Reads are fast and predictable because a key sits in exactly one place, but writes may require random disk I/O and page splits.

An LSM-tree-based engine (log-structured merge-tree) buffers writes in an in-memory table (a memtable), then flushes them as immutable sorted files (SSTables) on disk. A background compaction process merges these files to reclaim space and keep reads bounded.

Because a key can exist in several SSTables, a read may check multiple files; engines mitigate this with in-memory summaries and Bloom filters that quickly rule out files that cannot contain the key.

These engines are what make Database Indexing concrete: a B-tree index updates in place on every write, while an LSM index absorbs writes cheaply and pays later during compaction.

LSM-trees accept read amplification (checking several files) and background compaction cost in exchange for very high, sequential write throughput. B-trees accept write amplification and random I/O in exchange for stable, low-latency reads. Neither wins outright - the right engine follows whether the workload is write-heavy ingest or read-heavy transactions.

• When a system is write-heavy (metrics, events, chat history), reach for an LSM-backed store and say why.
• For read-heavy transactional workloads, a B-tree engine is the safer default.
• Mentioning compaction and Bloom filters signals real depth on LSM internals.