Bitter - A C++ Library for reading and writing bit fields

We are pleased to announce the first release of our new open source lightweight C++ library Bitter, which is a library for writing bit/byte fields into a data container. Here I will present our motivation for creating Bitter and go through a simple example for writing and reading bit/byte fields using Bitter.

Motivation

During the development of our communication protocol Score, we have had to work with the design and implementation of data packet headers. A part of this discipline is to read and write fields in a protocol header, this includes reading and writing fields which have an arbitrary size of bits ranging from 1 to N bits. However, in C++ there are no data type supporting data fields with a size smaller than 1 byte, unless one were to use a boolean to represent a single bit, bit field.

The solution to the above is to place, a data field of size N in a “container” data type of size M. Where the data type with size M, is the smallest data type which can contain the data field of size N. For example a data field with a size of 7 bits, should be placed in a container with the size of a byte. This however, presents an additional problem, because if we have a package header of size X and want to set two data fields in the package header, where the first field has size Y = 1/4 X and the second field has size Z = 3/4 X. Then if the combined size of the data types used for containing the two data fields exceeds X, we will be unable to set the fields of the header in a trivial manner.

But there already exist a solution to this problem, which is to use bit shifting and masking. This means that some data type of size X should be able to contain the two data fields of size Y and Z, such that Y + Z <= X. To implement this, we need to be aware of three things: A data fields offset in the container, the size of the data field such that a correct data max can be generated, and lastly the size of prior fields in the container, such that a bit shift is performed correctly as to not overwrite already set data bits in a container.

So utilizing shifting and masking is a possible solution. However, from our perspective there are problems with this approach. It is time consuming to write the code for shifting and masking, it is very error prone code and need extensive verification to ensure correctness, and the approach add unwanted complexity to the source code. So how to solved this? Well we had a couple of requirements for a solution we would accept, the solution should be reusable, encapsulated the complexity and if possible make it transparent for a developer that shifting and masking is being used. Based on these requirements, we started looking for different solutions to the problem and found a few solutions, but each had a problem either with license, how well the solution was document or how well encapsulation was utilized in the solution. Therefore we decided to develop Bitter and publish it under the BSD license, to make it available for all.

Now that you are aware of the motivation for creating Bitter, let me introduce you to Bitter with a simple example.

Simple Example

In the Bitter GitHub repository you will find the examples folder containing a simple writer and simple reader example, these are the two examples I will walk through and comment along the way.

But first a bit about the structure of bitter and how to use it. When you first download the Bitter project, you will see three folders: src, test, and examples. For usage src is the important one as it contains the implementation of Bitter itself, here you will also notice that Bitter is a header only library. test contains unit tests for Bitter and are used to ensure the correctness of bitter. Lastly examples contains a couple of examples for the usage of Bitter.

Writer Example

In the Bitter src folder you will find a couple of header files, of these we will only be using writer.hpp and reader.hpp and surprisingly to create a bit field writer, we will be using the writer.hpp.

Let us assume we will write a set of bit fields with following size and in the following order, 1 bit, 7, 8 and 16 bits, which have a combined size of 32 bits, by such the logical data container is a uint32_t, so we initialize our writer in the following manner

1
auto writer = bitter::writer<uint32_t, 1, 7, 8, 16>();

Now let us try to write the first field of size 1. For me this is a boolean, so either true or false, or 1 or 0. So let us write it as it is a boolean

1
2
boolean first = true;
writer.field<0>(first);

The type of the field is inferred by the value passed to the write method, so with a boolean we could have simplified the expression to writer.field<0>(true);. A note I would like to make is that the template parameter, is the index of the field, we wish to write and as can be seen fields are 0 index as is the C++ way. Now let us write the next field, which have a size of 7 bits. We will do this in the following manner

1
2
uint8_t value = 32U;
writer.field<1>(value);

So what do we observe here? First we observe a ”shortcoming” of C++, as stated in the motivation we need to wrap the 7 bit, bit field in a bigger container, in this case a unit8_t. But we also expose the transparency and encapsulation of Bitter because we only have to provide the index of the field and its value to place the field correctly in the data container. Behind the scenes Bitter uses the size of the field, we provided when initializing of the writer and the index we provide in the field method, hiding the fact that we are using bit shifting and masking to achieve this. Now let us continue with the third field, as it has a size of 8 bit, we can reuse the value variable we created above and simply write:

1
2
value = 128U;
write.field<2>(value);

Both of the prior calls could have been made into one-liners, for the third field we could have written write.field<2>(128U);. But with the next field I will explain why we have not done this. So let us continue with the final bit field, with a size of 16 bit. As with the prior fields Bitter handles shifting behind the scenes and we end up with the statement:

1
2
uint16_t larger_value = 2050U;
write.field<3>(value);

However, here we reach a limitation with type inference in C++. Because this call could have been written as writer.field<3>(2050U), due to uint16_t being the closest data type able of containing the unsigned integer 2050. But if the value of the third field had been 128 or below we would not have been able to just make the call writer.field<3>(128U); as type inference would determine the value to have the data type unit8_t. In the example we have choose to be very explicit about data types, but we could have used either a static_cast or a C-style cast. So the call could have been written as writer.field<3>(static_cast<uint32_t>(128U)): or writer.field<3>((uint32_t) 128U):.

Now let us check the data has been written to the uint32_t data container as we suspect. Now after shifting the values to the correct locations in the data container, the value of the uint32_t should be 0x8028041U denote in base 16. So we extract the data from the writer and assert the value.

1
2
auto data = writer.data();
assert(data == 0x8028041U);

If the assert passes, Bitter has done as we expected and we would be able to go on with data. So why not try to read it back and determine if the reader works, as expected?

Reader Example

Now as with the writer, we initialize the reader with the data type of the container and the field sizes in order. But unlike the writer, we also provide an input value, which is the data container we wish to read from so that the initialization statement becomes.

1
2
uint32_t data = 0x8028041U;
auto reader = bitter::reader<uint32_t, 1, 7, 8, 16>(data);

Before we get started, with actually reading, I will explain how we are going to check if the data is as expected and how we read data. We check the correctness of data by using asserts which means that assert(field_value == expected_value) must pass. Next when we call the field method on a reader, we are returned an instance of small class in Bitter called bit_field. This class provide us the ability to read the field with the read_as method, which takes a data type as template parameter and returns the value of the field, in a data container corresponding to the provided template parameter. It is not possible to provide a data type, which is not large enough to contain the field, as this is checked with a static_assert at compile time. Now that this is in order let us continue with reading from the data provided.

We read the first field with a size of 1 bit and again I will interpret this as a boolean. So we write.

1
2
auto first_field = reader.field<0>.read_as<bool>();
assert(first_field = true);

And so we continued, with the following fields. Again to emphasize, the second field has the size of 7 bits and therefore needs to be retrieved in a uint8_t as shown below.

1
2
3
4
5
6
7
8
auto second_field = reader.field<1>().read_as<uint8_t>();
assert(second_field == 32U);

auto third_field = reader.field<2>().read_as<uint8_t>();
assert(third_field == 128U);

auto fourth_field = reader.field<3>().read_as<uint16_t>();;
assert(fourth_field == 2050U);

So if all the assert statements are passed, we can say that Bitter has executed the commands as expected and our read data is valid. This is a very simple example just to showcase Bitter, but it could be utilized in combination with different transport protocol headers, file headers or something else.

Supported Containers

Now a note about what data types Bitter supports as containers. Bitter support uint8_t, uint16_t, uint32_t and uint64_t as data containers and accepts fields with a size of 64 bits or smaller. I hope you liked this small introduction to Bitter and if you want to investigate it further or hopefully use Bitter. You can find it on GitHub and it is freely available under the BSD license.

Thank you for reading.