failing compilation with musl

[Licenser]

Hi,
I'm trying to build an application using snmalloc in with a statically linked musl libc instead of the gnu libc, during that we've been running in a few problems.

I am not sure 100% where the issue orriginates, from my research it seems to be related to the usage of C++ and global destructors.

The error boils down to:

/usr/local/bin/../lib/gcc/x86_64-linux-musl/9.2.0/../../../../x86_64-linux-musl/bin/ld: /target/x86_64-unknown-linux-musl/debug/deps/tremor-17be01d5a5f619ac: hidden symbol `__dso_handle' isn't defined

To make it easy I've created a minimal reproduction with nothing but snmalloc-rs and a mainline function:

https://github.com/Licenser/snmalloc-musl

The error can be triggered by simply running make, it will build a musl docker image and then try to compile the application in it.

[darach]

At'ing @mjp41 for visibility in case similar musl issues occur with other snmalloc bindings in other PLs

[mjp41]

@davidchisnall any thoughts?

[davidchisnall]

__dso_handle is a magic symbol that's expected to be present with hidden visibility in every ELF shared library. It's passed to the C++ runtime function that handles destructors so that, when a library is unloaded all of the destructors can be run (and all of the thread-local destructors can be skipped if the library has been unloaded). It's usually provided by the CSU parts and might be missing with MUSL for statically linked binaries. I don't know why MUSL doesn't define it, because the C++ ABI requires that it exists, even for static linking.

Last time I looked at the musl, it didn't correctly handle library unloading even with shared libraries. It also doesn't implement __cxa_thread_atexit_impl or __cxa_thread_atexit at all, so the C++ runtime library has to fall back to using pthread_setspecific or similar for thread-local cleanup. I don't know what the implication of that is for snmalloc: It doesn't look as if musl needs to call free during thread teardown after it runs the thread-local destructors, so it's probably fine.

It is probably fine to just define something like this for static linking:

__attribute__((__visibility__((hidden)))
char __dso_handle;

You might actually be able to get away with a weak symbol: it doesn't matter if it resolves to 0, because the C++ runtime won't do anything with it other than equality comparisons.

It would be nice if more libc implementations had the explicit 'destroy malloc state for this thread' hook that FreeBSD libc provides.

[Licenser]

So I added

#[no_mangle]
pub static __dso_handle: u8 = 0;

to the mainline, that resolves the compilation issue but causes it to segfault:

(lldb) run
Process 547117 launched: '/home/heinz/Projects/Other/snmalloc-musl/target/x86_64-unknown-linux-musl/debug/snmalloc-musl' (x86_64)
Process 547117 stopped
* thread #1, name = 'snmalloc-musl', stop reason = signal SIGSEGV: invalid address (fault address: 0x7ff6)
    frame #0: 0x0000000000007ff6
error: memory read failed for 0x7e00
(lldb) bt
* thread #1, name = 'snmalloc-musl', stop reason = signal SIGSEGV: invalid address (fault address: 0x7ff6)
  * frame #0: 0x0000000000007ff6
    frame #1: 0x00007fffe7ddf0bf snmalloc-musl`void* snmalloc::Allocator<&(snmalloc::needs_initialisation(void*)), &(snmalloc::init_thread_allocator(snmalloc::function_ref<void* (void*)>)), snmalloc::MemoryProviderStateMixin<snmalloc::PALLinux>, snmalloc::DefaultChunkMap<snmalloc::GlobalPagemapTemplate<snmalloc::FlatPagemap<20ul, unsigned char> > >, true>::small_alloc_first_alloc<(snmalloc::ZeroMem)0, (snmalloc::AllowReserve)1>(unsigned long, unsigned long) at threadalloc.h:248:72
    frame #2: 0x00007fffe7ddf098 snmalloc-musl`void* snmalloc::Allocator<&(snmalloc::needs_initialisation(void*)), &(snmalloc::init_thread_allocator(snmalloc::function_ref<void* (void*)>)), snmalloc::MemoryProviderStateMixin<snmalloc::PALLinux>, snmalloc::DefaultChunkMap<snmalloc::GlobalPagemapTemplate<snmalloc::FlatPagemap<20ul, unsigned char> > >, true>::small_alloc_first_alloc<(snmalloc::ZeroMem)0, (snmalloc::AllowReserve)1>(unsigned long, unsigned long) at threadalloc.h:292
    frame #3: 0x00007fffe7ddf098 snmalloc-musl`void* snmalloc::Allocator<&(snmalloc::needs_initialisation(void*)), &(snmalloc::init_thread_allocator(snmalloc::function_ref<void* (void*)>)), snmalloc::MemoryProviderStateMixin<snmalloc::PALLinux>, snmalloc::DefaultChunkMap<snmalloc::GlobalPagemapTemplate<snmalloc::FlatPagemap<20ul, unsigned char> > >, true>::small_alloc_first_alloc<(this=<unavailable>, sizeclass=0, size=4)0, (snmalloc::AllowReserve)1>(unsigned long, unsigned long) at alloc.h:1142
    frame #4: 0x00007fffe7d8e684 snmalloc-musl`__rg_alloc [inlined] _$LT$snmalloc_rs..SnMalloc$u20$as$u20$core..alloc..global..GlobalAlloc$GT$::alloc::h0b33f8bdc654991c(self=0x00007fffe7de1870, layout=Layout @ 0x00007fffffffd650) at lib.rs:47:9
    frame #5: 0x00007fffe7d8e65d snmalloc-musl`__rg_alloc(arg0=4, arg1=1) at main.rs:2
    frame #6: 0x00007fffe7da3ea5 snmalloc-musl`std::rt::lang_start_internal::h53bf2c2c6c9e97dc [inlined] alloc::alloc::alloc::hc943e1a883a741c4 at alloc.rs:84:14
    frame #7: 0x00007fffe7da3e95 snmalloc-musl`std::rt::lang_start_internal::h53bf2c2c6c9e97dc [inlined] alloc::alloc::Global::alloc_impl::h2bac69f5a7e8918b at alloc.rs:164
    frame #8: 0x00007fffe7da3e95 snmalloc-musl`std::rt::lang_start_internal::h53bf2c2c6c9e97dc [inlined] _$LT$alloc..alloc..Global$u20$as$u20$core..alloc..AllocRef$GT$::alloc::h7513b9caadea9287 at alloc.rs:224
    frame #9: 0x00007fffe7da3e95 snmalloc-musl`std::rt::lang_start_internal::h53bf2c2c6c9e97dc [inlined] alloc::raw_vec::RawVec$LT$T$C$A$GT$::allocate_in::hf472df56c401029c at raw_vec.rs:189
    frame #10: 0x00007fffe7da3e95 snmalloc-musl`std::rt::lang_start_internal::h53bf2c2c6c9e97dc [inlined] alloc::raw_vec::RawVec$LT$T$C$A$GT$::with_capacity_in::ha57ededbd232e7a3 at raw_vec.rs:130
    frame #11: 0x00007fffe7da3e95 snmalloc-musl`std::rt::lang_start_internal::h53bf2c2c6c9e97dc [inlined] alloc::raw_vec::RawVec$LT$T$GT$::with_capacity::h8e52d47795fd752d at raw_vec.rs:93
    frame #12: 0x00007fffe7da3e95 snmalloc-musl`std::rt::lang_start_internal::h53bf2c2c6c9e97dc [inlined] alloc::vec::Vec$LT$T$GT$::with_capacity::h7fca68cb5cfe8e92 at vec.rs:363
    frame #13: 0x00007fffe7da3e95 snmalloc-musl`std::rt::lang_start_internal::h53bf2c2c6c9e97dc [inlined] alloc::slice::hack::to_vec::h118ebe2f38ae973e at slice.rs:159
    frame #14: 0x00007fffe7da3e95 snmalloc-musl`std::rt::lang_start_internal::h53bf2c2c6c9e97dc [inlined] alloc::slice::_$LT$impl$u20$$u5b$T$u5d$$GT$::to_vec::h6bb26fd0925afb64 at slice.rs:395
    frame #15: 0x00007fffe7da3e95 snmalloc-musl`std::rt::lang_start_internal::h53bf2c2c6c9e97dc [inlined] alloc::slice::_$LT$impl$u20$alloc..borrow..ToOwned$u20$for$u20$$u5b$T$u5d$$GT$::to_owned::h206c5471a834f4cd at slice.rs:728
    frame #16: 0x00007fffe7da3e95 snmalloc-musl`std::rt::lang_start_internal::h53bf2c2c6c9e97dc [inlined] alloc::str::_$LT$impl$u20$alloc..borrow..ToOwned$u20$for$u20$str$GT$::to_owned::h9d7e3d52bbc9568d at str.rs:205
    frame #17: 0x00007fffe7da3e95 snmalloc-musl`std::rt::lang_start_internal::h53bf2c2c6c9e97dc at rt.rs:44
    frame #18: 0x00007fffe7d8e597 snmalloc-musl`std::rt::lang_start::h41f2d223c97aece7(main=(snmalloc-musl`snmalloc_musl::main::hbdf7eca5d9ede0a9 at main.rs:7), argc=1, argv=0x00007fffffffd808) at rt.rs:65:5
    frame #19: 0x00007fffe7d8e85a snmalloc-musl`main + 42
    frame #20: 0x00007fffe7dc6afb snmalloc-musl`libc_start_main_stage2 + 39
    frame #21: 0x00007fffe7d8e097 snmalloc-musl`_start + 22

[davidchisnall]

It looks as if this is in the registration of the TLS destructor for OnDestruct. Is it possible for you to break on __cxa_thread_atexit and see what fails? The crash, unfortunately, looks as if something has been tail called and so some of the interesting stack frame is gone. You may need to compile musl and the C++ runtime library (libsupc++ / libc++abi / libcxxrt) in debug mode as well.

[mjp41]

I've spent a lot of effort making sure things get tail called ;-)

[davidchisnall]

I think this is inside the C++ runtime, not yours.

[mjp41]

It looks like this would be the first place OnDestruct is initialised as this is the main thread, so nothing else will have hit this yet. But are probably going to need a debug C++ rt and musl.

[mjp41]

Can we build the snmalloc-rs crate against musl?

[SchrodingerZhu]

but I remember rust itself relies on \_\_cxa\_thread\_atexit to handle the thread\_local dtor (see std library) . Did it specialize the behavior for MUSL? \-------- Original Message --------

On Jan 27, 2021, 9:55 PM, David Chisnall < ***@***.***> wrote: `__dso_handle` is a magic symbol that's expected to be present with hidden visibility in every ELF shared library. It's passed to the C++ runtime function that handles destructors so that, when a library is unloaded all of the destructors can be run (and all of the thread-local destructors can be skipped if the library has been unloaded). It's usually provided by the CSU parts and might be missing with MUSL for statically linked binaries. I don't know why MUSL doesn't define it, because the C++ ABI requires that it exists, even for static linking. Last time I looked at the musl, it [didn't correctly handle library unloading][didn_t correctly handle library unloading] even with shared libraries. It also doesn't implement `__cxa_thread_atexit_impl` or `__cxa_thread_atexit` at all, so the C++ runtime library has to fall back to using `pthread_setspecific` or similar for thread-local cleanup. I don't know what the implication of that is for snmalloc: It doesn't look as if musl needs to call `free` during thread teardown after it runs the thread-local destructors, so it's probably fine. It is *probably* fine to just define something like this for static linking: ``` __attribute__((__visibility__((hidden))) char __dso_handle; ``` You might actually be able to get away with a weak symbol: it doesn't matter if it resolves to 0, because the C++ runtime won't do anything with it other than equality comparisons. It would be nice if more libc implementations had the explicit 'destroy malloc state for this thread' hook that FreeBSD libc provides. — You are receiving this because you are subscribed to this thread. Reply to this email directly, [view it on GitHub][], or [unsubscribe][].![AEZNMJPUJWLGLPOGG37WKVTS4ALOXA5CNFSM4WVGVTC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOFXFVRGQ.gif][]

[didn_t correctly handle library unloading]: https://github.com/bminor/musl/blob/3ab2a4e02682df1382955071919d8aa3c3ec40d4/src/exit/atexit.c#L39 [view it on GitHub]: #102 (comment) [unsubscribe]: https://github.com/notifications/unsubscribe-auth/AEZNMJOPQ6JGKC4ALJHJSUDS4ALOXANCNFSM4WVGVTCQ [AEZNMJPUJWLGLPOGG37WKVTS4ALOXA5CNFSM4WVGVTC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOFXFVRGQ.gif]: https://github.com/notifications/beacon/AEZNMJPUJWLGLPOGG37WKVTS4ALOXA5CNFSM4WVGVTC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOFXFVRGQ.gif

[SchrodingerZhu]

This is how rust handles the issue. I don't see any differences has been done to musl? Or it just fallbacks?

[Licenser]

It looks as if this is in the registration of the TLS destructor for OnDestruct. Is it possible for you to break on __cxa_thread_atexit and see what fails? The crash, unfortunately, looks as if something has been tail called and so some of the interesting stack frame is gone. You may need to compile musl and the C++ runtime library (libsupc++ / libc++abi / libcxxrt) in debug mode as well.

Out of the box, sadly that isn't defined it seems, lldb doesn't find that breakpoint.

Can we build the snmalloc-rs crate against musl?

Yap I created a branch hat includes the bits needed for musl https://github.com/Licenser/snmalloc-rs/tree/musl

cargo install cross
cross test --target x86_64-unknown-linux-musl

runs the tests with musl and has the same results as the example repo

[davidchisnall]

Is it possible to link this with a debug build of musl and whatever C++ runtime you need (or can you install packages for these with separate debug info, if they're dynamically linked)? The bug seems to be in the interaction between clang-generated code from standard C++ language constructs, calling standard Itanium C++ runtime functions, and their implementations in the C++ runtime and libc.

[SchrodingerZhu]

Thou it may be irrelevant, but in my build script I required libstdc++ to be dynamic explicitly. I should have it fixed.