Data Layout

Benji works with two different categories of data: The metadata is stored in a database backend and describes how the blocks fit together to form a backup version. The actual blocks are saved in one or more object storages.

Database Backend

The database backend is responsible for managing all metadata.

The database backend relies on SQLAlchemy, a Python ORM which works with a huge number of DBMS, e.g. MySQL, PostgreSQL, SQLite3 and Oracle.

Benji has been developed and tested with PostgreSQL and SQLite3, so they are the recommended database engines and you may encounter problems with other databases. Patches to support other databases are welcome of course!

For Benji’s purposes, you may use either PostgreSQL or SQLite3. For a single workstation’s backup with 10 to 20 versions, SQLite3 is perfectly suitable. However you will benefit from PostgreSQL’s performance and stability when doing hundreds of versions with terabytes of backup data. A distributed installation of Benji requires PostgreSQL to work and it is also the recommended DBMS for production deployments.

You configure the location of your database with the databaseEngine directive. Please consult the SQLAlchemy documentation for options and format.

All timestamps in the database are in UTC.


Document all tables

Block Storages

The block storage backends are pluggable and there are currently three different implementations:

  • file: File based storage

  • s3: S3 compatible storage like AWS S3, Google Storage, Ceph’s RADOS Gateway or Minio

  • b2: Backblaze’s B2 Cloud Storage


Document information about the actual data layout, encryption, compression and mention metadata accompanying objects.