c++ - boost::thread::join() crash trying to reference destroyed thread_info

Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.
Learn more about Collectives
Teams
Q&A for work
Connect and share knowledge within a single location that is structured and easy to search.
Learn more about Teams
Edit: I've created a self-contained app that causes the problem. It requires normally over a 1000 iterations of the loop (creation/run/join of threads), sometimes not hitting the crash until several thousand iterations:
#include <boost/thread.hpp>
static void do_nothing() {}
int main() {
    int thread_count = 0;
    while (true) {
        thread_count++;
        boost::thread t1(boost::bind(&do_nothing));
        if (t1.joinable()) {
            t1.join();
And here is the address sanitizer dump after it catches use-of-freed-memory:
=================================================================
==96437==ERROR: AddressSanitizer: heap-use-after-free on address 0x058526f4 at pc 0x000a22cb bp 0xbffff4a8 sp 0xbffff4a4
WRITE of size 4 at 0x058526f4 thread T0
atos(96439,0x100357380) malloc: enabling scribbling to detect mods to free blocks
    #0 0xa22ca in boost::detail::atomic_decrement(int _Atomic*) sp_counted_base_clang.hpp:36
    #1 0xa21be in boost::detail::sp_counted_base::release() sp_counted_base_clang.hpp:115
    #2 0xa2157 in boost::detail::shared_count::~shared_count() shared_count.hpp:473
    #3 0xa115b in boost::detail::shared_count::~shared_count() shared_count.hpp:472
    #4 0x1ae8e63 in boost::thread::join_noexcept() shared_ptr.hpp:779
    #5 0x989a8c in boost::thread::join() thread.hpp:766
    #6 0x989366 in main main.cpp
0x058526f4 is located 4 bytes inside of 16-byte region [0x058526f0,0x05852700)
freed by thread T0 here:
    #0 0x35ca20d in wrap__ZdlPv (libclang_rt.asan_osx_dynamic.dylib:i386+0x6520d)
    #1 0x9ab89c in boost::detail::sp_counted_impl_p<boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(), boost::_bi::list0> > >::~sp_counted_impl_p() sp_counted_impl.hpp:53
    #2 0xa1dd1 in boost::detail::sp_counted_base::destroy() sp_counted_base_clang.hpp:97
    #3 0xa23e7 in boost::detail::sp_counted_base::weak_release() sp_counted_base_clang.hpp:131
    #4 0xa2262 in boost::detail::sp_counted_base::release() sp_counted_base_clang.hpp:118
    #5 0xa2157 in boost::detail::shared_count::~shared_count() shared_count.hpp:473
    #6 0xa115b in boost::detail::shared_count::~shared_count() shared_count.hpp:472
    #7 0x1ae8e56 in boost::thread::join_noexcept() shared_ptr.hpp:779
    #8 0x989a8c in boost::thread::join() thread.hpp:766
    #9 0x989366 in main main.cpp
previously allocated by thread T0 here:
    #0 0x35c9c0d in wrap__Znwm (libclang_rt.asan_osx_dynamic.dylib:i386+0x64c0d)
    #1 0x9ab357 in boost::detail::shared_count::shared_count<boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(), boost::_bi::list0> > >(boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(), boost::_bi::list0> >*) shared_count.hpp:137
    #2 0x9ab214 in boost::detail::shared_count::shared_count<boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(), boost::_bi::list0> > >(boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(), boost::_bi::list0> >*) shared_count.hpp:132
    #3 0x9aaf88 in void boost::detail::sp_pointer_construct<boost::detail::thread_data_base, boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(), boost::_bi::list0> > >(boost::shared_ptr<boost::detail::thread_data_base>*, boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(), boost::_bi::list0> >*, boost::detail::shared_count&) shared_ptr.hpp:284
    #4 0x9aae3b in boost::shared_ptr<boost::detail::thread_data_base>::shared_ptr<boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(), boost::_bi::list0> > >(boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(), boost::_bi::list0> >*) shared_ptr.hpp:362
    #5 0x99c804 in boost::shared_ptr<boost::detail::thread_data_base>::shared_ptr<boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(), boost::_bi::list0> > >(boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(), boost::_bi::list0> >*) shared_ptr.hpp:361
    #6 0x99c407 in boost::shared_ptr<boost::detail::thread_data_base> boost::thread::make_thread_info<boost::_bi::bind_t<void, void (*)(), boost::_bi::list0> >(boost::_bi::bind_t<void, void (*)(), boost::_bi::list0>, boost::disable_if_c<is_same<boost::decay<boost::_bi::bind_t<void, void (*)(), boost::_bi::list0> >::type, boost::thread>::value, boost::thread::dummy*>::type) thread.hpp:229
    #7 0x99c120 in boost::thread::thread<boost::_bi::bind_t<void, void (*)(), boost::_bi::list0> >(boost::_bi::bind_t<void, void (*)(), boost::_bi::list0>, boost::disable_if_c<boost::thread_detail::is_rv<boost::_bi::bind_t<void, void (*)(), boost::_bi::list0> >::value, boost::thread::dummy*>::type) thread.hpp:299
    #8 0x989826 in boost::thread::thread<boost::_bi::bind_t<void, void (*)(), boost::_bi::list0> >(boost::_bi::bind_t<void, void (*)(), boost::_bi::list0>, boost::disable_if_c<boost::thread_detail::is_rv<boost::_bi::bind_t<void, void (*)(), boost::_bi::list0> >::value, boost::thread::dummy*>::type) thread.hpp:300
    #9 0x989366 in main main.cpp
SUMMARY: AddressSanitizer: heap-use-after-free sp_counted_base_clang.hpp:36 in boost::detail::atomic_decrement(int _Atomic*)
Shadow bytes around the buggy address:
  0x20b0a480: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd
  0x20b0a490: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd
  0x20b0a4a0: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd
  0x20b0a4b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x20b0a4c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x20b0a4d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa[fd]fd
  0x20b0a4e0: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd
  0x20b0a4f0: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd
  0x20b0a500: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd
  0x20b0a510: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd
  0x20b0a520: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
 =================================================================
 ==96437==ERROR: AddressSanitizer: heap-use-after-free on address 0x058526f4 at pc 0x000a22cb bp 0xbffff4a8 sp 0xbffff4a4
 WRITE of size 4 at 0x058526f4 thread T0
     #0 0xa22ca in boost::detail::atomic_decrement(int _Atomic*) sp_counted_base_clang.hpp:36
     #1 0xa21be in boost::detail::sp_counted_base::release() sp_counted_base_clang.hpp:115
     #2 0xa2157 in boost::detail::shared_count::~shared_count() shared_count.hpp:473
     #3 0xa115b in boost::detail::shared_count::~shared_count() shared_count.hpp:472
     #4 0x1ae8e63 in boost::thread::join_noexcept() shared_ptr.hpp:779
     #5 0x989a8c in boost::thread::join() thread.hpp:766
     #6 0x989366 in main main.cpp
 0x058526f4 is located 4 bytes inside of 16-byte region [0x058526f0,0x05852700)
 freed by thread T0 here:
     #0 0x35ca20d in wrap__ZdlPv (libclang_rt.asan_osx_dynamic.dylib:i386+0x6520d)
     #1 0x9ab89c in boost::detail::sp_counted_impl_p<boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(), boost::_bi::list0> > >::~sp_counted_impl_p() sp_counted_impl.hpp:53
     #2 0xa1dd1 in boost::detail::sp_counted_base::destroy() sp_counted_base_clang.hpp:97
     #3 0xa23e7 in boost::detail::sp_counted_base::weak_release() sp_counted_base_clang.hpp:131
     #4 0xa2262 in boost::detail::sp_counted_base::release() sp_counted_base_clang.hpp:118
     #5 0xa2157 in boost::detail::shared_count::~shared_count() shared_count.hpp:473
     #6 0xa115b in boost::detail::shared_count::~shared_count() shared_count.hpp:472
     #7 0x1ae8e56 in boost::thread::join_noexcept() shared_ptr.hpp:779
     #8 0x989a8c in boost::thread::join() thread.hpp:766
     #9 0x989366 in main main.cpp
 previously allocated by thread T0 here:
     #0 0x35c9c0d in wrap__Znwm (libclang_rt.asan_osx_dynamic.dylib:i386+0x64c0d)
     #1 0x9ab357 in boost::detail::shared_count::shared_count<boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(), boost::_bi::list0> > >(boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(), boost::_bi::list0> >*) shared_count.hpp:137
     #2 0x9ab214 in boost::detail::shared_count::shared_count<boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(), boost::_bi::list0> > >(boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(), boost::_bi::list0> >*) shared_count.hpp:132
     #3 0x9aaf88 in void boost::detail::sp_pointer_construct<boost::detail::thread_data_base, boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(), boost::_bi::list0> > >(boost::shared_ptr<boost::detail::thread_data_base>*, boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(), boost::_bi::list0> >*, boost::detail::shared_count&) shared_ptr.hpp:284
     #4 0x9aae3b in boost::shared_ptr<boost::detail::thread_data_base>::shared_ptr<boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(), boost::_bi::list0> > >(boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(), boost::_bi::list0> >*) shared_ptr.hpp:362
     #5 0x99c804 in boost::shared_ptr<boost::detail::thread_data_base>::shared_ptr<boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(), boost::_bi::list0> > >(boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(), boost::_bi::list0> >*) shared_ptr.hpp:361
     #6 0x99c407 in boost::shared_ptr<boost::detail::thread_data_base> boost::thread::make_thread_info<boost::_bi::bind_t<void, void (*)(), boost::_bi::list0> >(boost::_bi::bind_t<void, void (*)(), boost::_bi::list0>, boost::disable_if_c<is_same<boost::decay<boost::_bi::bind_t<void, void (*)(), boost::_bi::list0> >::type, boost::thread>::value, boost::thread::dummy*>::type) thread.hpp:229
     #7 0x99c120 in boost::thread::thread<boost::_bi::bind_t<void, void (*)(), boost::_bi::list0> >(boost::_bi::bind_t<void, void (*)(), boost::_bi::list0>, boost::disable_if_c<boost::thread_detail::is_rv<boost::_bi::bind_t<void, void (*)(), boost::_bi::list0> >::value, boost::thread::dummy*>::type) thread.hpp:299
     #8 0x989826 in boost::thread::thread<boost::_bi::bind_t<void, void (*)(), boost::_bi::list0> >(boost::_bi::bind_t<void, void (*)(), boost::_bi::list0>, boost::disable_if_c<boost::thread_detail::is_rv<boost::_bi::bind_t<void, void (*)(), boost::_bi::list0> >::value, boost::thread::dummy*>::type) thread.hpp:300
     #9 0x989366 in main main.cpp
 SUMMARY: AddressSanitizer: heap-use-after-free sp_counted_base_clang.hpp:36 in boost::detail::atomic_decrement(int _Atomic*)
 Shadow bytes around the buggy address:
   0x20b0a480: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd
   0x20b0a490: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd
   0x20b0a4a0: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd
   0x20b0a4b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
   0x20b0a4c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 =>0x20b0a4d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa[fd]fd
   0x20b0a4e0: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd
   0x20b0a4f0: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd
   0x20b0a500: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd
   0x20b0a510: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd
   0x20b0a520: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd
 Shadow byte legend (one shadow byte represents 8 application bytes):
   Addressable:           00
   Partially addressable: 01 02 03 04 05 06 07
   Heap left redzone:       fa
   Freed heap region:       fd
   Stack left redzone:      f1
   Stack mid redzone:       f2
   Stack right redzone:     f3
   Stack after return:      f5
   Stack use after scope:   f8
   Global redzone:          f9
   Global init order:       f6
   Poisoned by user:        f7
   Container overflow:      fc
   Array cookie:            ac
   Intra object redzone:    bb
   ASan internal:           fe
   Left alloca redzone:     ca
   Right alloca redzone:    cb
Build Environment: Boost 1.63 built under macOS LLVM 9.0 with libc++ (host app built with same). Host is running a debug version, so no compiler optimization. Boost was probably built with optimization on, but I'll have to look into that.
Anyone have any clue as to where I should look? TBH, I'm not sure what join_noexcept() is doing there with the shared_ptr resetting exactly – why is the necessary? I don't think this is a Boost bug, but I'm at a loss on where to look. The host app might be stomping on something of course, but I've looked into that extensively and haven't found anything yet. The crashes consistently occur when the join() is happening.
Original post before I created the minimal case example above: I'm seeing infrequent crashes when waiting for a communication thread to shutdown via join(). The crashes happen infrequently, but I can reproduce it with some stress testing after a few hours of opening/closing a communication thread ~3 times per second.
I captured the crash with debug memory options enabled (zombie blocks, etc.), and it is showing that the in:
thread::join_noexcept()
            if(thread_info==local_thread_info)
                thread_info.reset();
the conditional is True, so the reset() is performed resulting in the thread_data being destroyed with a stack that looks like:
detail::sp_counted_impl_p<boost::detail::thread_data<CThreadAdapter> >::~sp_counted_impl_p() at sp_counted_impl.hpp:53
detail::sp_counted_base::destroy() at sp_counted_base_clang.hpp:97
detail::sp_counted_base::weak_release() at sp_counted_base_clang.hpp:131
detail::sp_counted_base::release() at sp_counted_base_clang.hpp:118
detail::shared_count::~shared_count() at shared_count.hpp:473
detail::shared_count::~shared_count() at shared_count.hpp:472
shared_ptr<boost::detail::thread_data_base>::~shared_ptr() [inlined] at shared_ptr.hpp:779
shared_ptr<boost::detail::thread_data_base>::~shared_ptr() [inlined] at shared_ptr.hpp:779
shared_ptr<boost::detail::thread_data_base>::reset() [inlined] at shared_ptr.hpp:667
thread::join_noexcept() at thread.cpp:343
thread::join() at thread.hpp:766
(above is not the crash, just where the memory is deallocated that is later referenced)
The crash then occurs in the same join_noexcept() call after it is complete when it is destroying its locals. It appears it tries to access the freed thread_data (I think):
detail::atomic_decrement(int _Atomic*) at sp_counted_base_clang.hpp:36
detail::sp_counted_base::release() at sp_counted_base_clang.hpp:115
detail::shared_count::~shared_count() at shared_count.hpp:473
detail::shared_count::~shared_count() at shared_count.hpp:472
shared_ptr<boost::detail::thread_data_base>::~shared_ptr() [inlined] at shared_ptr.hpp:779
shared_ptr<boost::detail::thread_data_base>::~shared_ptr() [inlined] at shared_ptr.hpp:779
thread::join_noexcept() at thread.cpp:351
thread::join() at thread.hpp:766
                You need to reduce to a minimal self-contained example. Out on a limb, the problem is probably elsewhere. Maybe you can check that joinable() returns true.
– sehe
                Jul 20, 2018 at 13:22
                I added a self-contained example and address sanitizer dump to the top of the original post.
– mbbeme
                Jul 20, 2018 at 19:43
                With respect to the shared_ptr, it could be implementation detail behind thread_specific_ptr and/or boost::thread::at_thread_exit - but I'm guessing
– sehe
                Jul 20, 2018 at 19:58
                I don't know the cause of the bug, but I think you want to use scoped_thread to avoid this issue.
– Hitobat
                Jul 20, 2018 at 20:39
The added self-contained code is worth a thousand words.
It is clear that the bug is not in the driver code, and boost is very unlikely to blame. So that leaves 
Resource leaks
Undefined Behaviour due to ODR/ABI issues
Your information suggests that you've been extra careful to build the library and the test program with compatible compiler, libraries and flags. This sort of rules out ABI/ODR issues.
That brings me to resource leaks. It seems to me that conditionally joining risks not-joining a thread if there is any race around "joinable()". I wouldn't have thought this to be the case, but in your simple self-contained example you could see whether one of the following removes the crash:
remove the condition:
#include <boost/thread.hpp>
#include <iostream>
static void do_nothing() {}
int main() {
    uintmax_t thread_count = 0;
    while (++thread_count) {
        boost::thread(do_nothing).join();
    std::cout << "Done\n";
avoid a race (by delaying the do_nothing exit, or by using a synchronization primitive to signal the end of the thread). To be fair, this strikes me as a "workaround" that would indicate a library bug.
Okay, the problem turned out to be I was building the libboost_thread library with 
 BOOST_AC_USE_PTHREADS BOOST_SP_USE_PTHREADS defined. I'm not sure which one of those cause it, but one of them causes a crash in the thread destructor where the shared_ptr to the thread data is destroyed twice. Neither of those flags should be needed, however, when targeting macOS. So by removing them (see my pull request here) the crash is avoided.
It does seem like a bug though that trying to use pthread_mutex_t for atomic count and shared_ptr locking causes a crash on macOS (note it happens less than 0.08% of the time but if you tight loop like the code snippet above you'll hit it). The problem is probably rarely hit since the default behavior for building boost on macOS doesn't define those constants.
        Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.