Memory efficiency benchmark

In the following test, we filled Dragonfly and Redis with ~5GB of data
using debug populate 5000000 key 1024 command. Then we started sending the update traffic with memtier and kicked off the snapshotting with the
“bgsave” command. The following figure demonstrates clearly how both servers behave in terms of memory efficiency.

Dragonfly was 30% more memory efficient than Redis at the idle state.
It also did not show any visible memory increase during the snapshot phase.
Meanwhile, Redis reached almost x3 memory increase at peak compared to Dragonfly.
Dragonfly also finished the snapshot much faster, just a few seconds after it started.
For more info about memory efficiency in Dragonfly see dashtable doc

Running the server

Dragonfly runs on linux. It uses relatively new linux specific io-uring API
for I/O, hence it requires Linux version 5.10 or later.
Debian/Bullseye, Ubuntu 20.04.4 or later fit these requirements.

With docker:

docker pull docker.dragonflydb.io/dragonflydb/dragonfly && 
docker tag docker.dragonflydb.io/dragonflydb/dragonfly dragonfly

docker run --network=host --ulimit memlock=-1 --rm dragonfly

redis-cli PING  # redis-cli can be installed with "apt install -y redis-tools"

You need --ulimit memlock=-1 because some Linux distros configure the default memlock limit for containers as 64m and Dragonfly requires more.

Releases

We maintain binary releases for x86 and arm64 architectures. You will need to install libunwind8 lib to run the binaries.

Building from source

You need to install dependencies in order to build on Ubuntu 20.04 or later:

git clone --recursive https://github.com/dragonflydb/dragonfly && cd dragonfly

# to install dependencies
sudo apt install ninja-build libunwind-dev libboost-fiber-dev libssl-dev 
     autoconf-archive libtool

# Configure the build
./helio/blaze.sh -release

# Build
cd build-opt && ninja dragonfly

# Run
./dragonfly --alsologtostderr

Configuration

Dragonfly supports common redis arguments where applicable.
For example, you can run: dragonfly --requirepass=foo --bind localhost.

Dragonfly currently supports the following Redis-specific arguments:

• port
• bind
• requirepass
• maxmemory
• dir – by default, dragonfly docker uses /data folder for snapshotting.
  You can use -v docker option to map it to your host folder.
• dbfilename

In addition, it has Dragonfly specific arguments options:

• memcache_port – to enable memcached compatible API on this port. Disabled by default.
• keys_output_limit – maximum number of returned keys in keys command. Default is 8192.
  keys is a dangerous command. we truncate its result to avoid blowup in memory when fetching too many keys.
• dbnum – maximum number of supported databases for select.
• cache_mode – see Cache section below.

for more options like logs management or tls support, run dragonfly --help.

Roadmap and status

Currently Dragonfly supports ~130 Redis commands and all memcache commands besides cas.
We are almost on par with Redis 2.8 API. Our first milestone will be to stabilize basic
functionality and reach API parity with Redis 2.8 and Memcached APIs.
If you see that a command you need, is not implemented yet, please open an issue.

The next milestone will be implementing H/A with redis -> dragonfly and
dragonfly<->dragonfly replication.

For dragonfly-native replication we are planning to design a distributed log format that will
support order of magnitude higher speeds when replicating.

After replication and failover feature we will continue with other Redis commands from
APIs 3,4 and 5.

Initial release

API 1.0

• String family
  • SET
  • SETNX
  • GET
  • DECR
  • INCR
  • DECRBY
  • GETSET
  • INCRBY
  • MGET
  • MSET
  • MSETNX
  • SUBSTR
• Generic family
  • DEL
  • ECHO
  • EXISTS
  • EXPIRE
  • EXPIREAT
  • KEYS
  • PING
  • RENAME
  • RENAMENX
  • SELECT
  • TTL
  • TYPE
  • SORT
• Server Family
  • AUTH
  • QUIT
  • DBSIZE
  • BGSAVE
  • SAVE
  • DEBUG
  • EXEC
  • FLUSHALL
  • FLUSHDB
  • INFO
  • MULTI
  • SHUTDOWN
  • LASTSAVE
  • SLAVEOF/REPLICAOF
  • SYNC
• Set Family
  • SADD
  • SCARD
  • SDIFF
  • SDIFFSTORE
  • SINTER
  • SINTERSTORE
  • SISMEMBER
  • SMOVE
  • SPOP
  • SRANDMEMBER
  • SREM
  • SMEMBERS
  • SUNION
  • SUNIONSTORE
• List Family
  • LINDEX
  • LLEN
  • LPOP
  • LPUSH
  • LRANGE
  • LREM
  • LSET
  • LTRIM
  • RPOP
  • RPOPLPUSH
  • RPUSH
• SortedSet Family
  • ZADD
  • ZCARD
  • ZINCRBY
  • ZRANGE
  • ZRANGEBYSCORE
  • ZREM
  • ZREMRANGEBYSCORE
  • ZREVRANGE
  • ZSCORE
• Other
  • BGREWRITEAOF
  • MONITOR
  • RANDOMKEY
  • MOVE

API 2.0

• List Family
  • BLPOP
  • BRPOP
  • BRPOPLPUSH
  • LINSERT
  • LPUSHX
  • RPUSHX
• String Family
  • SETEX
  • APPEND
  • PREPEND (dragonfly specific)
  • BITCOUNT
  • BITFIELD
  • BITOP
  • BITPOS
  • GETBIT
  • GETRANGE
  • INCRBYFLOAT
  • PSETEX
  • SETBIT
  • SETRANGE
  • STRLEN
• HashSet Family
  • HSET
  • HMSET
  • HDEL
  • HEXISTS
  • HGET
  • HMGET
  • HLEN
  • HINCRBY
  • HINCRBYFLOAT
  • HGETALL
  • HKEYS
  • HSETNX
  • HVALS
  • HSCAN
• PubSub family
  • PUBLISH
  • PUBSUB
  • PUBSUB CHANNELS
  • SUBSCRIBE
  • UNSUBSCRIBE
  • PSUBSCRIBE
  • PUNSUBSCRIBE
• Server Family
  • WATCH
  • UNWATCH
  • DISCARD
  • CLIENT LIST/SETNAME
  • CLIENT KILL/UNPAUSE/PAUSE/GETNAME/REPLY/TRACKINGINFO
  • COMMAND
  • COMMAND COUNT
  • COMMAND GETKEYS/INFO
  • CONFIG GET/REWRITE/SET/RESETSTAT
  • MIGRATE
  • ROLE
  • SLOWLOG
  • PSYNC
  • TIME
  • LATENCY…
• Generic Family
  • SCAN
  • PEXPIREAT
  • PEXPIRE
  • DUMP
  • EVAL
  • EVALSHA
  • OBJECT
  • PERSIST
  • PTTL
  • RESTORE
  • SCRIPT LOAD/EXISTS
  • SCRIPT DEBUG/KILL/FLUSH
• Set Family
  • SSCAN
• Sorted Set Family
  • ZCOUNT
  • ZINTERSTORE
  • ZLEXCOUNT
  • ZRANGEBYLEX
  • ZRANK
  • ZREMRANGEBYLEX
  • ZREMRANGEBYRANK
  • ZREVRANGEBYSCORE
  • ZREVRANK
  • ZUNIONSTORE
  • ZSCAN
• HYPERLOGLOG Family
  • PFADD
  • PFCOUNT
  • PFMERGE

Memchache API

• set
• get
• replace
• add
• stats (partial)
• append
• prepend
• delete
• flush_all
• incr
• decr
• version
• quit

Some commands were implemented as decorators along the way:

• ROLE (2.8) decorator as master.
• UNLINK (4.0) decorator for DEL command
• BGSAVE (decorator for save)
• FUNCTION FLUSH (does nothing)

Milestone – H/A

Implement leader/follower replication (PSYNC/REPLICAOF/…).

Milestone – “Maturity”

APIs 3,4,5 without cluster support, without modules and without memory introspection commands. Also
without geo commands and without support for keyspace notifications, without streams.
Probably design config support. Overall – few dozens commands…
Probably implement cluster-API decorators to allow cluster-configured clients to connect to a
single instance.

Next milestones will be determined along the way.

Design decisions

Novel cache design

Dragonfly has a single unified adaptive caching alg