top of page
  • Hunter Allen

Zero-Copy Strings and Bounded Vectors

Updated: Nov 30, 2022


Introduction

Apex.OS has a specialized transport mechanism that allows the developer to publish messages through shared memory without requiring copies. However, until now, many data types for which this would be most useful, such as large images and point clouds, were incompatible with this zero-copy transport.


The reason lies in the memory layout of these data types. Generally, images and point cloud data are stored as variable length sequences (which are mapped to apex::BoundedVector) and have frame IDs (human readable names for coordinate systems). The entire type must be contiguous in memory for a data type to be compatible with zero-copy transport. Variable length data structures such as Bounded Vectors or strings merely hold a pointer to their data, which is stored in memory allocated separately, and does not fulfill this requirement.



In Apex.OS 2.1.0 , we added extended zero-copy support to include these non-trivial data structures by addressing this memory layout issue and introducing an IDL annotation to allow the developer to enable this new capability explicitly.


apex::detail::StorageBase

In order to enable this expanded zero-copy support, explicit control over the memory allocated by our container types is required so that it is possible to construct them in a way where all of their memory is contiguous. We did not want to create entirely new container types, so we created a new class: apex::detail::StorageBase, which exists to encapsulate memory allocations for our containers. The entire function of this class is to construct a buffer to store elements of type T, either in a method that is contiguous within itself (T[]) or externally as before (T *).


The class has three template parameters:

  1. BUFFER_SIZE: represents the number of elements that can be stored in the buffer

  2. T: The type of elements to be stored in the buffer

  3. Alloc: The allocator with which to allocate the buffer.

The memory layout of the container is entirely determined via the choice of Alloc. For this use case, in particular, we have defined apex::detail::StackAllocator, which is used to specify that memory should be allocated contiguously. This is done by creating a specialization of apex::detail::StorageBase that contains a contiguous array rather than a pointer to external memory.


The class also implements operator[], the move constructor and move assignment operator, and two helper functions to allow the parent structure to construct and destruct elements at a specific index (construct_at and destruct_at).


Changes to apex::BoundedVector

With the new StorageBase container implemented, we simply declare the storage type within the BoundedVector implementation as follows.

using storage_type = apex::detail::StorageBase<UpperBound, value_type, allocator_type>;

Then, with minimal refactoring (mostly changing pointer arithmetic to accessing/addressing via operator[]), we have a container that can use either storage strategy, does not add a new template parameter, and does not break the API.


Changes to apex::string

Since the apex::string implementation uses calloc and free to allocate and deallocate, we implemented a special allocator, apex::detail::CAlloc<T> to mimic this behavior in the new data structure as closely as possible.

The apex::string implementation already had the ability to swap out the storage container, so the most significant change to these structures was simply swapping out the old storage structure and using the new StorageBase class.


Changes to Message Generation

Since the choice of storage method is entirely based on the type of allocator, we need some way to inject this into the message as it is being generated. We could simply use this layout as the default, but this could cause severe issues for users if they did something like declare a message on the stack.


PointCloud2 msg;


This would be perfectly fine previously (as the points would be stored as a pointer, and the massive array containing the points would be heap allocated); however, if the message were to be declared with apex::detail::StackAllocator, the data entry (declared as sequence<uint8, 7500000> data) would be an incredible 7.5 MB, which almost exhausts the default stack size by itself (which is a little more than 8 MB). Any existing code that had a similar declaration would undoubtedly feel the consequences of making this behavior the new default, so it’s best to avoid this.


Instead, we introduce a new IDL annotation, @flat, to enable this feature explicitly.

module perf_zerocopy_msg { module msg { @flat struct FlatSequencesBounded { sequence<boolean, 10> bool_value; string<10> string_value; sequence<string<10>, 10> strings_value; }; }; };


The above message will generate a @flat message that uses apex::detail::StackAllocator. This annotation also marks the message as self-contained so that Apex.Middleware recognizes the message as something it can transport via the zero-copy API.

Interestingly, once serialization enters the picture, these data structures behave similarly . First, the middleware will only transport the occupied contents of the strings or bounded vectors, so there is no performance penalty over the wire. Then, when deserializing the message, the middleware can deserialize the data to either flat or non-flat data structures as required. As such, there are no interoperability problems between flat and non-flat endpoints as long as the message is serialized.


Future Apex.OS releases will enable marking messages in sensor_msgs, std_msgs, etc., as @flat to make this feature more readily available.


Performance Results



The graph above shows the latency for bounded sequences of various sizes. As seen above, zero-copy with the @flat annotation outperforms shared memory transport quite dramatically with larger message sizes, with a reduction from roughly 3.5ms down to 0.5 ms for 4-megabyte sequences.


As the performance data above shows, enabling the use of zero-copy can bring huge benefits to applications that process large amounts of data. By introducing the @flat annotation, we have made it possible to support virtually all message types with zero-copy while maintaining API compatibility. Additionally, since @flat can be added as needed, existing code will continue to work without requiring any changes, allowing developers to selectively enable it for types that will benefit most from zero copy while leaving less critical message types alone.



If you are interested in Apex.AI products for your projects, contact us.

We’re always looking for awesome people who want to join our worldwide team!

Visit our careers page for a list of open positions.





784 views

Recent Posts

See All
bottom of page