Returning view in c++20

[rsarwar87]

hi

i have a design on an untrascale with 4 gb ram. i have partitioned it to be half reserved for DMA intake. Problem is when I try to send data that is say, 1.2 GB, the system seems to crash. that is because as returns i am limited to array tuple. i was hoping to send as quickly as possible without needing to copy the memory anywhere using span.

auto& get_adc1_data() {
         std::span<uint32_t, n_desc * n_pts> buffer_view{
             ram_s2mm_1.get_ptr<uint32_t, mem::ocm_s2mm1_range>()
         };
          return buffer_view;
    }

i ran into two problems, it seems

1. incorrect use of static assert. this is because span is supported in g++10 upwards. and this was just a compiler error:

239 | static_assert(req_buff_size <= cmd.payload.size(), "Buffer size too small"); | ~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~ tmp/examples/teb808/adc_lvds_dma_ts4//server/interface_DataMover.cpp:239:33: error: 'cmd' is not a constant expression

cmd.payload.size() is not static,. so i changed it to . not sure if i should crash the serverd, or let it pass.

index 905cd7cf..38bce905 100644
--- a/server/__init__.py
+++ b/server/__init__.py
@@ -243,7 +243,9 @@ def parser_generator(driver, operation):
 
     if not has_vector:
         print_required_buff_size(lines, packs)
-        lines.append('    static_assert(req_buff_size <= cmd.payload.size(), "Buffer size too small");\n\n');
+        lines.append('    if (req_buff_size > cmd.payload.size()) {')
+        lines.append('        throw std::runtime_error("Buffer size too small");')
+        lines.append('    }')
 
     for idx, pack in enumerate(packs):
         if pack['family'] == 'scalar':

2. looking at the command DynamicSerializer, it seems its still using a buffer to copy things over. is it possible to implement support for view whitout needing to use the buffer. what may stop it.

[tvanderbruggen]

Hi,

1. You can simply use:

static_assert(req_buff_size <= CMD_PAYLOAD_BUFFER_LEN, "Buffer size too small");

2. The way the protocol works is that we get a header followed by the data (payload):

// |      RESERVED     | dev_id  |  op_id  |             payload_size              |   payload
// |  0 |  1 |  2 |  3 |  4 |  5 |  6 |  7 |  8 |  9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | ...

At some point you need to copy the actual data after the header, so the minimum you can do is one copy here
(there is a second copy done by the OS at the send() syscall).

You could imagine an API which would provide a pointer to the payload position in the buffer:

// |      RESERVED     | dev_id  |  op_id  |             payload_size              |   payload
// |  0 |  1 |  2 |  3 |  4 |  5 |  6 |  7 |  8 |  9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | ...
//                                                                                 ^--- Give this pointer

Then the user could build the payload in place and avoid the copy.
But in your case where the data are already in some other memory that won't help...

NB: The span is already kind of what we do when we send a std::array from read_array():

auto p = get_ptr<std::array<T, N>, offset>(block_idx);

Here we are simply cast the pointer into a std::array type.
That's just a compile time manipulation provide data type + array size safety, read_array() does not copy anything, copy is only performed once at buffer generation.

[rsarwar87]

excellent point of the using std::array<T, N> from get_ptr. ill try it, it should help reduce having to make one copy instead of two.

regarding the API does the memory have to be continious though? would this not be possible:

#include <ranges>
#include <span>

auto combined = std::views::join(std::array{std::span(*header), std::span(*payload)});// payload as returned from the fuction

so we need to modify the send buffer in the Session i think?

[tvanderbruggen]

The standard BSD socket write API is requiring a pointer and a count so the buffer as to be continuous.
std::views::join is just a convenient way to get iterators for use with STL algorithms, making it simpler to write code when copies are not required.

In your case you could statically write a header in your dedicated RAM and run the DMA at the header offset.
Then you would have to hardwired the call to this buffer in the server core, but still there will be a user-space to kernel-space copy.

One could try the MSG_ZEROCOPY flag, for example if the payload size is larger than 10 kB...

[rsarwar87]

I noticed in this commit:

b5777a3#diff-155895127268933ce78b04f0a5fe82e8861d55e15bc190852a89172909c51e13

that zero-copy spans are now supported in v1.

Is there a list of feature changes for v1? The serverd component seems substantially different from previous versions. From what I can tell, it looks like the 2023 branch can still be built with Vivado 2025, so I assume the FPGA components haven’t changed significantly? But the make files and the config files all have changed substantially.

I’m trying to understand how difficult it would be to back-port the serverd from v1 into the 0.x branch. At UKAEA we have a large number of boards already brought under support, and it would be unfortunate to have to redo all of the ports and re-validate the hardware again.

Also, I’m curious about the rationale behind splitting config.yml into separate Makefiles. Was there a specific design reason that gave an advantage in the new build chain?

[tvanderbruggen]

Hi,

To be precise the zero copy has been implemented in this commits:

• 6c7b2aa
• e8ec346

Support for std::span transfers has been added, but the zero copy works also for std::array and std::vector.
Note that every memory is not eligible for zero-copy: it must be a CMA (Contiguous Memory Allocator) region.
Also I did not manage to saturate the TCP link, I went to ~60 MiB/s, and according to this
the max. rate on a Zynq 7020 is 700 Mbps (~84 MiB/s) still below the theoretical 115 MiB/s for TCP,
it seems the interconnect is the bottleneck.
So zerocopy is not a huge win...

Regarding config.yml splitting.
We've moved away the files dependency listings from the config.yml into a config.mk because it's more logical to have make handling the files.
This not only removes a lot of boilerplate python code to inject the dependencies into the make system,
but also provides a lot of flexibility to organize the build.
For example up to now we've fully implemented the drivers into the hpp files but you may want to split the implementation into a cpp file (to drag less dependencies for faster build) now it is easy to set this extra dependency in the build system (see for example examples/alpha250/fft/config.mk.

Once you've moved away the files from the config.yml it remains the fields memory and parameters so we've decided to rename it memory.yml. From the project configuration side it's not fundamentally different from what was done before.
Maybe try to migrate one project and tell us if you hit a difficulty ?