How to pass on arrays of raw bytes?

[rackaracka]

I'm implementing a python module that allows to create a Msg data type. It takes
integers and an array of bytes. When I call the toString method of my type, it
prints all the values:

T: 17, L: 13, V: [2,3,5,7,11,13,17,19,23,127,-128,-127,0], S: [17,0,0,0,13,2,3,5,7,11,13,17,19,23,127,-128,-127,0]

V is the actual value and I've got a method that returns that value. However it returns
a shared_prt<int8_t[]), so I created a lambda that shall return a vector holding those
byte values:

.def ( "getValue", [] (const rav::Msg &m){
    std::shared_ptr<int8_t[]> data = m.getValue();
    std::vector<int8_t> _vector;
    for(uint i; i < m.getLength(); ++i)
    {
        _vector.push_back(data[i]);
    }
    return _vector;
}, py::return_value_policy::move )

Calling it prints an empty vector:
print(f"value: {msg.getValue()}")

value: []

OK, maybe I did make some mistakes initialising it. So I added a few cout's:

.def ( "getValue", [] (const rav::Msg &m){
    std::shared_ptr<int8_t[]> data = m.getValue();
    std::vector<int8_t> _vector;
    for(uint i; i < m.getLength(); ++i)
    {
        std::cerr << ", " << std::to_string(data[i]);
        _vector.push_back(data[i]);
    }
    std::cerr << std::endl;
    std::cerr << "vector size: " << std::to_string(_vector.size()) << std::endl;
    for(int8_t _v : _vector) {
        std::cerr << " , " << std::to_string(_v);
    }
    std::cerr << std::endl;


    return _vector;
}, py::return_value_policy::move )

The method execution prints:

, 1, 1, 1, 1, 11, 13, 17, 19, 23, 127, -128, -127, 1
vector size: 13
, 2 , 3 , 5 , 7 , 11 , 13 , 17 , 19 , 23 , 127 , -128 , -127 , 0

And the script returns:
print(f"value: {msg.getValue()}")

value: [2, 3, 5, 7, 11, 13, 17, 19, 23, 127, -128, -127, 0]

I thought, maybe the code without using the variable gets optimised away (using -O2 as compiler flag).
However changing it to -O0 did not make any difference.

I'm new to C++ and Python, so I'm clearly doing it wrong. What is the correct way
of passing an array of bytes (that may contain 0 values) to a C++ constructor
or method or returning the such a byte array?

[gistrec]

The root bug is this loop:

for(uint i; i < m.getLength(); ++i)

i is never initialized. In C++, a local variable like uint i; has an indeterminate value. If that value happens to be greater than or equal to m.getLength(), the loop body never runs and the returned vector stays empty

Adding std::cerr changes the generated code / stack layout enough to mask the bug in that particular build, which is why it may look like an optimization issue

Fix:

for (uint i = 0; i < m.getLength(); ++i)

That said, if the goal is to return raw bytes to Python, py::bytes is usually the simpler option:

.def("getValue", [](const rav::Msg &m) {
    auto data = m.getValue();
    return py::bytes(reinterpret_cast<const char *>(data.get()),
                     static_cast<pybind11::ssize_t>(m.getLength()));
})

Python will receive a bytes object. Indexing a bytes object returns integers in the range 0..255

If you need signed int8_t values rather than raw bytes, consider returning py::array_t<int8_t> or building a typed list/vector explicitly instead

[rackaracka]

Thanks for your hints. I managed to get it done with the
numpy array:

Taking it as an argument to a constructor:

std::shared_ptr<int8_t[]> convert(py::array_t<int8_t> value)
{
    auto buffer = value.request();
    int8_t* ptr = static_cast<int8_t*>(buffer.ptr);
    return std::shared_ptr<int8_t[]>(ptr);
}

rav::Msg create(int8_t type, uint32_t length, py::array_t<int8_t> value) {
    rav::Msg msg { type, length, convert(value) };
    return msg;
}

...
.def(py::init<>(&create))

and returning it like:

.def ( "getValue", [] (const rav::Msg &m){
    std::shared_ptr<int8_t[]> data = m.getValue();
    // create a numpy array
    py::array_t<int8_t> arr(m.getLength());
    // get the pointer to its internal array
    auto buffer = arr.request();
    int8_t* ptr = static_cast<int8_t*>(buffer.ptr);
    // copy the message values into it
    for(uint i = 0; i < m.getLength(); ++i)
    {
        ptr[i] =  data.get()[i];
    }
    return arr;
})

It looks still a bit clumsy but at least it works.