Hexi is a lightweight, header-only C++23 library for safely handling binary data from arbitrary sources (but primarily network data). It sits somewhere between manually memcpying bytes from network buffers and full-blown serialisation libraries.
The design goals are ease of use, safety when dealing with untrusted data, a reasonable level of flexibility, and keeping overhead to a minimum.
What Hexi doesn’t offer: versioning, conversion between different formats, handling of text-based formats, unloading the dishwasher.
Incorporating Hexi into your project is simple! The easiest way is to simply copy hexi.h from single_include into your own project. If you’d rather only include what you use, you can add include to your include paths or incorporate it into your own CMake project with target_link_library. To build the unit tests, run CMake with ENABLE_TESTING.
Here’s what some libraries might call a very simple motivating example:
#include <hexi.h> #include <array> #include <vector> #include <cstddef> struct UserPacket { uint64_t user_id; uint64_t timestamp; std::array<uint8_t, 16> ipv6; }; auto deserialise(std::span<const char> network_buffer) { hexi::buffer_adaptor adaptor(network_buffer); // wrap the buffer hexi::binary_stream stream(adaptor); // create a binary stream // deserialise! UserPacket packet; stream >> packet; return packet; } auto serialise(const UserPacket& packet) { std::vector<uint8_t> buffer; hexi::buffer_adaptor adaptor(buffer); // wrap the buffer hexi::binary_stream stream(adaptor); // create a binary stream // serialise! stream << packet; return buffer; }
By default, Hexi will try to serialise basic structures such as our UserPacket if they meet requirements for being safe to directly copy the bytes. Now, for reasons of portability, it’s not recommended that you do things this way unless you’re positive that the data layout is identical on the system that wrote the data. Not to worry, this is easily solved. Plus, we didn’t do any error handling. All in good time.
The two classes you’ll primarily deal with are buffer_adaptor and binary_stream.
binary_stream takes a container as its argument and is used to do the reading and writing. It doesn’t know much about the details of the underlying container.
To support containers that weren’t written to be used with Hexi, buffer_adaptor is used as a wrapper that binary_stream can interface with. As with binary_stream, it also provides read and write operations but at a lower level.
buffer_adaptor can wrap any contiguous container or view that provides data and size member functions and op
13 Comments
gregschlom
Semi off-topic, but I just love the header image and the advice frog in the readme. Makes reading the documentation more fun and enjoyable.
nightowl_games
Wow that api looks fantastic! Bravo!
I'd like to read an even more thorough overview of how it works and all the gotchas before I'd consider using this 'in production' but the API looks very easy to use and very elegant.
EDIT: just hit the section on portability, seems like you would always have to use that API, yeah? I feel like when you are writing network code you simply have to make it portable from the get-go. I guess I'm always thinking about having it run on client machines.
klaussilveira
It's been a while since I saw a new library with such a clean interface. Congrats!
codedokode
It doesn't look like zero-copy though in this example:
That is at least one copy.
Jyaif
Your lib requires manually creating both a serializing and deserializing function.
If the functions are out of sync, bad things happen.
Consider copying Cereal, which solves this problem by requiring you to create a single templated function ( https://uscilab.github.io/cereal/ )
codedokode
By the way I looked through the code, and had to read about metaprogramming in C++. I wonder why is it so complicated? For example, why constraints like std::is_integral are represented by structs. Doesn't make much sense to me. A function wouldn't be better here?
connicpu
I know it's a convention since the inception of the language, but the operator overload abuse of the bitshift operator still makes me sad every time I see it :(
423642643
[dead]
abcd_f
In the same vein, but without needing to create separate de- and serialize functions:
https://github.com/eliasdaler/MetaStuff
Another take on the same idea with even simpler interface:
https://github.com/apankrat/cpp-serializer
huhtenberg
What are the exact constraints on the struct contents, i.e. what is it that your library can't serialize?
I tried adding std::string to the UserPacket (from the README)
and the compilation fails – https://onlinegdb.com/B_RJd5Uws
Labo333
Fun! It reminds me of my own attempt at this: https://github.com/louisabraham/ubuf
It can generate efficient JS and C++ from a simple YAML file.
rybosome
Lovely API, great work on that.
curtisszmania
[dead]