What's causing these crashes after cross-compiling?

Question

I'm trying to cross-compile a large library (TensorFlow) using gcc on Ubuntu. I've installed the g++-arm-linux-gnueabihf toolchain, and was able to successfully build my binary. The process I'm using to build is documented here: https://github.com/petewarden4prs/tensorflow/tree/master/tensorflow/contrib/makefile#raspberry-pi

Initially I hit an error that pthreading was disabled ("Enable multithreading to use std::thread: Operation not permitted") when I tried to run the resulting executable on my Pi 3. I recompiled with -pthread enabled as a compile option, and now the program crashes seemingly at random with segmentation faults. Running it in gdb, they often seem related to free() being called with bad pointers, and the call stacks seem corrupt, so I'm assuming there's some memory mismatch happening.

Does anyone have suggestions on things I can try to track down what's going wrong here?

Here are some more details from my Pi:

pi@raspberrypi ~ $ uname -a
Linux raspberrypi 4.1.19-v7+ #858 SMP Tue Mar 15 15:56:00 GMT 2016 armv7l GNU/Linux
pi@raspberrypi ~ $ file benchmark 
benchmark: ELF 32-bit LSB executable, ARM, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=0x5043384f5d0003f8074b07dfdd38cdc20315143f, not stripped

Here's an example of a typical session in gdb:

[New Thread 0x76cf5450 (LWP 6011)]
*** glibc detected *** /home/pi/benchmark: free(): invalid pointer: 0x018e2e89 ***

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x76cf5450 (LWP 6011)]
0x76f98e40 in std::string::c_str() const () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
(gdb) thread apply all bt

Thread 2 (Thread 0x76cf5450 (LWP 6011)):
#0  0x76f98e40 in std::string::c_str() const () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
#1  0x00bad996 in tensorflow::thread::ThreadPool::Impl::WorkerLoop() ()
#2  0x00bad5de in tensorflow::thread::ThreadPool::Impl::Impl(tensorflow::Env*, tensorflow::ThreadOptions const&, std::string const&, int)::{lambda()#1}::operator()() const ()
#3  0x00badec2 in std::_Function_handler<void (), tensorflow::thread::ThreadPool::Impl::Impl(tensorflow::Env*, tensorflow::ThreadOptions const&, std::string const&, int)::{lambda()#1}>::_M_invoke(std::_Any_data const&) ()
#4  0x0029aaf4 in std::function<void ()>::operator()() const ()
#5  0x00b53e1e in _ZNSt12_Bind_simpleIFSt8functionIFvvEEvEE9_M_invokeIJEEEvSt12_Index_tupleIJXspT_EEE ()
#6  0x00b53d90 in std::_Bind_simple<std::function<void ()> ()>::operator()() ()
#7  0x00b53d4a in std::thread::_Impl<std::_Bind_simple<std::function<void ()> ()> >::_M_run() ()
#8  0x76f91848 in ?? () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
#9  0x76f91848 in ?? () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 1 (Thread 0x76ff6000 (LWP 6010)):
#0  0x76dfc61c in ?? () from /lib/arm-linux-gnueabihf/libc.so.6
#1  0x76fff048 in ?? () from /lib/ld-linux-armhf.so.3
Cannot access memory at address 0x158
#2  0x76fff048 in ?? () from /lib/ld-linux-armhf.so.3
Cannot access memory at address 0x158
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Answer 1

The easiest way to a binary compatible cross-compilation is to install the toolchain used by Raspbian developers. It can be found here. It's essential to use this toolchain if you want to build the kernel and drivers, as kernel objects require perfect ABI compatibility, but having perfect compatibility won't hurt if you're building userspace binaries as well.

According to documentation, this toolchain is compatible with current Ubuntu, both 32-bit and 64-bit.

Answer 2

I was getting a pure virtual method called exception when cross-compiling. @JeremyBarnes's answer did not quite work for me. Instead I used:

-U__GCC_HAVE_SYNC_COMPARE_AND_SWAP_1 -U__GCC_HAVE_SYNC_COMPARE_AND_SWAP_2 -U__GCC_HAVE_SYNC_COMPARE_AND_SWAP_8

Explaination:

As @JeremyBarnes pointed out, to ensure ABI compatibility of your application with the installed stdc++, they both need to be compiled with the same SYNC flags.

On Raspbian:

$ g++ -dM -E - < /dev/null | grep SYNC
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4 1

Without the fix on dockcross/linux-armv6 and dockcross/linux-armv7:

$ g++ -dM -E - < /dev/null | grep SYNC
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_1 1
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_2 1
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4 1
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8 1

With the fix on dockcross/linux-armv6 and dockcross/linux-armv7:

$ g++ -U__GCC_HAVE_SYNC_COMPARE_AND_SWAP_1 -U__GCC_HAVE_SYNC_COMPARE_AND_SWAP_2  -E - < /dev/null | grep SYNC
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4 1

Answer 3

FWIW, this can be fixed by adding -D__GCC_HAVE_SYNC_COMPARE_AND_SWAP_1 -D__GCC_HAVE_SYNC_COMPARE_AND_SWAP_2 -D__GCC_HAVE_SYNC_COMPARE_AND_SWAP_8 to the compiler flags.

Why? In /usr/include/c++/4.{8,9}/bits/concurrency.h, the default lock policy depends upon these defines:

#if (defined(__GCC_HAVE_SYNC_COMPARE_AND_SWAP_2) \
     && defined(__GCC_HAVE_SYNC_COMPARE_AND_SWAP_4))

The ABI of a shared pointer depends upon how these flags are defined, as it inherits from a base class which uses a default template argument for the locking policy. Hence, changing these flags changes the layout (because it changes the base class layout) of the std::shared_ptr<...> objects in the standard C++ library.

In the compiler that comes with the Pi, with which Raspbian was built, they are set as follows:

g++ -dM -E - < /dev/null | grep SYNC
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4 1

This is sensible for the Pi 1, but is a big shame for the Pi 3 which can quite happily use atomic shared pointers.

On Ubuntu, they are set up like this:

arm-linux-gnueabihf-g++ -dM -E - < /dev/null | grep SYNC
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_1 1
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_2 1
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4 1
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8 1

The command line flags above reset them to how they are by default on the Pi.

Cross compiling is worth it; Tensorflow is already slow to build on a beefy server; must take an incredibly long time to build on the Pi!