3 posts tagged with "git"

In the previous post, we followed the white rabbit down into Git's .git folder. We understood: Git uses zlib's deflate algorithm to compress objects.

The Problem: Each range of zlib-compressed data depends on reading all the data before it. A 1Gb sqlite database compressed with zlib must read the whole file to access a single row[^1].

In this article we're going to build a mental model on how zlib compresses and we're going to use Z_FULL_FLUSH to compress data in a way which is compatible with Git and allows us to random access data in objects stored in git.

Join me on a jump right into the rabbit hole on what we find when we look closer at the .git folder. We'll explore how git stores your code in loose objects and discover a flag in a 30yr old library used by git that could let us run databases inside git.

In the upcoming series of blogposts I'm going to share the results of my research over the last years.

The result: A way to enable random read and write access in git's storage, driving a version controlled fs that allows to run versioned sqlite database on top of git's internal storage.