Stores small-sized, immutable snapshots of your data in an append-only manner. It facilitates querying and reconstructing the entire history as well as easy audits.
Download ZIP | Join us on Discord | Community Forum | Documentation | Architecture & Concepts
Working on your first Pull Request? You can learn how from this free series How to Contribute to an Open Source Project on GitHub and another tutorial: How YOU can contribute to OSS, a beginners guide
“Remember that you’re lucky, even if you don’t think you are because there’s always something that you can be thankful for.” – Esther Grace Earl (http://tswgo.org)
We want to build the database system together with you. Help us and become a maintainer yourself. Why? You may like the software and want to help us. Furthermore, you’ll learn a lot. You may want to fix a bug or add a feature. Do you want to add an awesome project to your portfolio? Do you want to grow your network?… All of this are valid reasons besides probably many more: Collaborating on Open Source Software
SirixDB appends data to an indexed log file without the need of a WAL. It can be embedded and used as a library from your favorite language on the JVM to store and query data locally or by using a simple CLI. An asynchronous HTTP server, which adds the core and query modules as dependencies, can interact with SirixDB over the network using Keycloak for authentication/authorization. One file stores the data with all revisions and possibly secondary indexes. A second file stores offsets into the file to quickly search for a revision by a given timestamp using an in-memory binary search. Furthermore, a few maintenance files exist, which store the configuration of a resource and the definitions of secondary indexes (if any are configured). Other JSON files keep track of changes in delta files if enabled.
It currently supports the storage and (time travel) querying of XML and JSON data in its binary encoding, tailored to support versioning. The index structures and the whole storage engine has been written from scratch to support versioning natively. We might also implement storing and querying other data formats as relational data.
SirixDB uses a huge persistent (in the functional sense) tree of tries, wherein the committed snapshots share unchanged pages and even common records in changed pages. The system only stores page fragments during a copy-on-write out-of-place operation instead of full pages during a commit to reduce write-amplification. During read operations, the system reads the page fragments in parallel to reconstruct an in-memory page (thus, a fast, random access storage device as a PCIe SSD is best suited or even byte-addressable storage as Intel DC optane memory shortly — as SirixDB stores fine granular cache-size (not page) aligned modifications in a single file.
Please consider sponsoring our Open Source work if you like the project.
Note: Let us know if you’d like to build a brand-new frontend with, for instance Svelte, D3.js, and Typescript.
Discuss it in the Community Forum.
Table of contents
- Keeping All Versions of Your Data By Sharing Structure
- JSONiq examples
- SirixDB Features
- Getting Started
- Getting Help
- Contributors
- License
Keeping All Versions of Your Data By Sharing Structure
We could write a lot about why keeping all states of your data in a storage system is of great value. In a nutshell, it’s all about looking at the evolution of your data, finding trends, doing audits, and implementing efficient undo-/redo-operations. The Wikipedia page has a bunch of examples. We recently also added use cases over here.
Our firm belief is that a temporal storage system must address the issues which arise from keeping past states way better than traditional approaches. Usually, storing time-varying, temporal data in database systems that do not support the storage thereof natively results in many unwanted hurdles. They waste storage space, query performance to retrieve past states of your data is not most ideal, and usually, temporal operations are missing altogether.
The DBS must store data so that storage space is used as effectively as possible while supporting the reconstruction of each revision, as the database saw it during the commits. All this should be handled in linear time, whether it’s the first revision or the most recent revision. Ideally, the query time of old/past revisions and the most recent revision should be in the same runtime complexity (logarithmic when querying for specific records).
SirixDB not only supports snapshot-based versioning on a record granular level through a novel versioning algorithm called sliding snapshot, but also time travel queries, efficient diffing between revisions, and storing semi-structured data.
Executing the following time-travel query on our binary JSON representation of Twitter sample data gives an initial impression of the possibilities:
The query opens a database/resource in a specific revision based on a timestamp (2019–04–13T16:24:27Z) and searches for all statuses, which have a created_at timestamp, which has to be greater than the 1st of February in 2018 and did not exist in the previous revision. . is a dereferencing operator used to dereference keys in JSON objects, array values can be accessed as shown looping over the values or through specifying an index, starting with zero: array[[0]] for instance, specifies the first value of the array. Brackit, our query processor, also supports Python-like array slices to simplify tasks.
JSONiq examples
To verify changes in a node or its subtree, first, select the node in the revision and then
query for changes using our stored Merkle hash tree, which builds and updates hashes for each node and its subtree and checks the hashes with sdb:hash($item). The function jn:all-times delivers the node in all revisions in which it exists. jn:previous delivers
the node in the previous revision or an empty sequence if there’s none.
let $node := jn:doc('mycol.jn','mydoc.jn').fieldName[[1]] let $result := for $node-in-rev in jn:all-times($node) let $nodeInPreviousRevision := jn:previous($node-in-rev) return if ((not(exists($nodeInPreviousRevision))) or (sdb:hash($node-in-rev) ne sdb:hash($nodeInPreviousRevision))) then $node-in-rev else () return [ for $jsonItem in $result return { "node": $jsonItem, "revision": sdb:revision($jsonItem) } ]
Emit all diffs between the revisions in a JSON format:
We support easy updates as in
let $array := jn:doc('mycol.jn','mydoc.jn') return insert json {"bla":true} into $array at position 0
to insert a JSON object into a resource, whereas the root node is an array at the first position (0). The transaction is implicitly committed. Thus, a new revision is created, and the specific revision can be queried using a single third argument, either a simple integer ID or a timestamp. The following query issues a query on the first revision (thus without the changes).
jn:doc('mycol.jn','mydoc.jn',1)
Omitting the third argument opens the resource in the most recent revision, but you could, in this case, also specify revision number 2. You can also use a timestamp as in:
jn:open('mycol.jn','mydoc.jn',xs:dateTime('2022-03-01T00:00:00Z'))
A simple join (whereas joins are optimized in our query processor called Brackit):
(* first: store stores in a stores resource *) sdb:store('mycol.jn','stores',' [ { "store number" : 1, "state" : "MA" }, { "store number" : 2, "state" : "MA" }, { "store number" : 3, "state" : "CA" }, { "store number" : 4, "state" : "CA" } ]') (* second: store sales in a sales resource *) sdb:store('mycol.jn','sales',' [ { "product" : "broiler", "store number" : 1, "quantity" : 20 }, { "product" : "toaster", "store number" : 2, "quantity" : 100 }, { "product" : "toaster", "store number" : 2, "quantity" : 50 }, { "product" : "toaster", "store number" : 3, "quantity" : 50 }, { "product" : "blender", "store number" : 3, "quantity" : 100 }, { "product" : "blender", "store number" : 3, "quantity" : 150 }, { "product" : "socks", "store number" : 1, "quantity" : 500 }, { "product" : "socks", "store number" : 2, "quantity" : 10 }, { "product" : "shirt", "store number" : 3, "quantity" : 10 } ]') let $stores := jn:doc('mycol.jn','stores') let $sales := jn:doc('mycol.jn','sales') let $join := for $store in $stores, $sale in $sales where $store."store number" = $sale."store number" return { "nb" : $store."store number", "state" : $store.state, "sold" : $sale.product } return [$join]
SirixDB through Brackit also supports array slices. The start index is 0, the step is 1, and end index is 1 (exclusive) in the next query:
let $array := [{"foo": 0}, "bar", {"baz": true}] return $array[[0:1:1]]
The query returns the first object {"foo":0}.
With the function sdb:nodekey you can find out the internal unique node key of a node, which will never change. You for instance might be interested in which revision it has been removed. The following query uses the function sdb:select-item which, as the first argument needs a context node and, as the second argument the key of the item or node to select. jn:last-existing finds the most recent version and sdb:revision retrieves the revision number.
sdb:revision(jn:last-existing(sdb:select-item(jn:doc('mycol.jn','mydoc.jn',1), 26)))
Index types
SirixDB has three types of indexes along with a path summary tree, which is basically a tree of all distinct paths:
- name indexes, to index a set of object fields
- path indexes, to index a set of paths (or all paths in a resource)
- CAS indexes, so-called content-and-structure indexes, which index paths and typed values (for instance, all xs:integers). In this case, on the paths specified, only integer values are indexed on the path, but no other types
We base the indexes on the following serialization of three revisions of a very small SirixDB resource.
{
"sirix": [
{
"revisionNumber": 1,
"revision": {
"foo": [
"bar",
null,
2.33
],
"bar": {
"hello": "world",
"helloo": true
},
"baz": "hello",
"tada": [
{
"foo": "bar"
},
{
"baz": false
},
"boo",
{},
[]
]
}
},
{
"revisionNumber": 2,
"revision": {
"tadaaa": "todooo",
"foo": [
"bar",
null,
103
],
"bar": {
"hello": "world",
"helloo": true
},
"baz": "hello",
"tada": [
{
"foo": "bar"
},
{
"baz": false
},
"boo",
{},
[]
]
}
},
{
"revisionNumber": 3,
"revision": {
"tadaaa": "todooo",
"foo": [
"bar",
null,
23.76
],
"bar": {
"hello": "world",
"helloo": true
},
"baz": "hello",
"tada": [
{
"foo": "bar"
},
{
"baz": false
},
"boo",
{},
[
{
"foo": "bar"
}
]
]
}
}
]
}
let $doc := jn:doc('mycol.jn','mydoc.jn') let $stats := jn:create-name-index($doc, ('foo','bar')) return {"revision": sdb:commit($doc)}
The index is created for “foo” and “bar” object fields. You can query for “foo” fields as for instance:
let $doc := jn:doc('mycol.jn','mydoc.jn') let $nameIndexNumber := jn:find-name-index($doc, 'foo') for $node in jn:scan-name-index($doc, $nameIndexNumber, 'foo') order by sdb:revision($node), sdb:nodekey($node) return {"nodeKey": sdb:nodekey($node), "path": sdb:path($node), "revision": sdb:revision($node)}
Second, whole paths are indexable.
Thus, the following path index is applicable to both queries: .sirix[].revision.tada[].foo and .sirix[].revision.tada[][[4]].foo. Thus, essentially both foo nodes are indexed and the first child has to be fetched afterwards. For the second query also the array index 4 has to be checked if the indexed node is really on index 4.
let $doc := jn:doc('mycol.jn','mydoc.jn') let $stats := jn:create-path-index($doc, '/sirix/[]/revision/tada//[]/foo') return {"revision": sdb:commit($doc)}
The index might be scanned as follows:
let $doc := jn:doc('mycol.jn','mydoc.jn') let $pathIndexNumber := jn:find-path-index($doc, '/sirix/[]/revision/tada//[]/foo') for $node in jn:scan-path-index($doc, $pathIndexNumber, '/sirix/[]/revision/tada//[]/foo') order by sdb:revision($node), sdb:nodekey($node) return {"nodeKey": sdb:nodekey($node), "path": sdb:path($node)}
CAS indexes index a path plus the value. The value itself must be typed (so in this case we index only decimals on a path).
let $doc := jn:doc('mycol.jn','mydoc.jn') let $stats := jn:create-cas-index($doc, 'xs:decimal', '/sirix/[]/revision/foo/[]') return {"revision": sdb:commit($doc)}
We can do an index range-scan as for instance via the next query (2.33 and 100 are the min and max, the next two arguments are two booleans which denote if the min and max should be retrieved or if it’s >min and
let $doc := jn:doc('mycol.jn','mydoc.jn') let $casIndexNumber := jn:find-cas-index($doc, 'xs:decimal', '/sirix/[]/revision/foo/[]') for $node in jn:scan-cas-index-range($doc, $casIndexNumber, 2.33, 100, false(), true(), ()) order by sdb:revision($node), sdb:nodekey($node) return {"nodeKey": sdb:nodekey($node), "node": $node}
You can also create a CAS index on all string values on all paths (all object fields: //*; all arrays: //[]):
let $doc := jn:doc('mycol.jn','mydoc.jn') let $stats := jn:create-cas-index($doc,'xs:string',('//*','//[]')) return {"revision": sdb:commit($doc)}
To query for string values with a certain name (bar) on all paths (empty sequence ()):
let $doc := jn:doc('mycol.jn','mydoc.jn') let $casIndexNumber := jn:find-cas-index($doc, 'xs:string', '//*') for $node in jn:scan-cas-index($doc, $casIndexNumber, 'bar', '==', ()) order by sdb:revision($node), sdb:nodekey($node) return {"nodeKey": sdb:nodekey($node), "node": $node, "path": sdb:path(sdb:select-parent($node))}
The argument == means check for equality of the string. Other values that make more sense for integers, and decimals… are <, <=, >= and >.
SirixDB Features
SirixDB is a log-structured, temporal JSON and XML database system, which stores evolutionary data. It never overwrites any data on disk. Thus, we're able to restore and query the full revision history of a resource in the database.
Design Goals
Some of the most important core principles and design goals are:
- Embeddable
- Similar to SQLite and DucksDB, SirixDB is embeddable at its core. Other APIs as the non-blocking REST API are built on top.
- Minimize Storage Overhead
- SirixDB shares unchanged data pages as well as records between revisions, depending on a chosen versioning algorithm during the initial bootstrapping of a resource. SirixDB aims to balance read and writer performance in its default configuration.
- Concurrent
- SirixDB contains very few locks and aims to be as suitable for multithreaded systems as possible.
- Asynchronous
- Operations can happen independently; each transaction is bound to a specific revision and only one read/write transaction on a resource is permitted concurrently with N read-only transactions.
- Versioning/Revision history
- SirixDB stores a revision history of every resource in the database without imposing extra overhead. It uses a huge persistent, durable page tree for indexing revisions and data.
- Data integrity
- SirixDB, like ZFS, stores full checksums of the pages in the parent pages. That means that almost all data corruption can be detected upon reading in the future; we aim to partition and replicate databases in the future.
- Copy-on-write semantics
- Similarly to the file systems Btrfs and ZFS, SirixDB uses CoW semantics, meaning that SirixDB never overwrites data. Instead, database-page fragments are copied/written to a new location. SirixDB does not simply copy whole pages. Instead, it only copies changed records plus records, which fall out of a sliding window.
- Per revision and page versioning
- SirixDB does not only version on a per revision but also on a per page base. Thus, whenever we change a potentially small fraction
of records in a data page, it does not have to copy the whole page and write it to a new location on a disk or flash drive. Instead, we can specify one of several versioning strategies known from backup systems or a novel sliding snapshot algorithm during the creation of a database resource. The versioning type we specify is used by SirixDB to version data pages. - Guaranteed atomicity and consistency (without a WAL)
- The system will never enter an inconsistent state (unless there is hardware failure), meaning that unexpected power-off won't ever damage the system. This is accomplished without the overhead of a write-ahead log.
(WAL) - Log-structured and SSD friendly
- SirixDB batches writes and syncs everything sequentially to a flash drive
during commits. It never overwrites committed data.
Revision Histories
Keeping the revision history is one of the main features in
SirixDB. You can revert any revision to an earlier version or back up the system automatically without the overhead of copying. SirixDB only ever copies changed database pages and, depending on the versioning algorithm you chose during the creation of a database/resource, only page fragments, and ancestor index pages to create a new revision.
You can reconstruct every revision in O(n), where n
denotes the number of nodes in the revision. Binary search is used on
an in-memory (linked) map to load the revision, thus finding the
revision root page has an asymptotic runtime complexity of O(log
n), where n, in this case, is the number of stored
revisions.
Currently, SirixDB offers two built-in native data models, namely a
binary XML store and a JSON store.
Articles published on Medium:
