Got a coredump
Let’s say a daemon got a segmentation fault.
$ coredumpctl
TIME PID UID GID SIG COREFILE EXE SIZE
Tue 9999-99-99 00:00:00 PDT 12345 0 0 SIGSEGV present /usr/bin/bmcweb 20.7MDump the core and SCP it form the machine to your workstation.
$ coredumpctl dump 12345 > /tmp/bmcweb.corePrepare new workspace
Create a brand new workspace and copy require files over.
$ mkdir -p /var/gbmc/coredumpCompile GDB
Depending on your target machine, build the corresponding cross GDB.
$ bitbake gdb-cross-aarch64
# bitbake gdb-cross-arm for 32 bits machine
$ cp /var/gbmc/gbmc/build/platform/tmp/work/x86_64-linux/gdb-cross-aarch64/14.1/image/var/gbmc/gbmc/build/platform/tmp/work/x86_64-linux/gdb-cross-aarch64/14.1/recipe-sysroot-native/usr/bin/aarch64-openbmc-linux/aarch64-openbmc-linux-gdb /var/gbmc/coredump/
/var/gbmc/coredump$ ldd aarch64-openbmc-linux-gdb
linux-vdso.so.1 (0x00007fc766b0d000)
libexpat.so.1 => /var/gbmc/gbmc/build/platform/tmp/work/x86_64-linux/gdb-cross-aarch64/14.1/recipe-sysroot-native/usr/lib/libexpat.so.1 (0x00007fc766add000)
libreadline.so.8 => /var/gbmc/gbmc/build/platform/tmp/work/x86_64-linux/gdb-cross-aarch64/14.1/recipe-sysroot-native/usr/lib/libreadline.so.8 (0x00007fc766a86000)
libz.so.1 => /var/gbmc/gbmc/build/platform/tmp/work/x86_64-linux/gdb-cross-aarch64/14.1/recipe-sysroot-native/usr/lib/libz.so.1 (0x00007fc766a6c000)
libzstd.so.1 => /var/gbmc/gbmc/build/platform/tmp/work/x86_64-linux/gdb-cross-aarch64/14.1/recipe-sysroot-native/usr/lib/libzstd.so.1 (0x00007fc765f3e000)
libtinfo.so.5 => /var/gbmc/gbmc/build/platform/tmp/work/x86_64-linux/gdb-cross-aarch64/14.1/recipe-sysroot-native/usr/lib/libtinfo.so.5 (0x00007fc765f0f000)
libpython3.12.so.1.0 => /var/gbmc/gbmc/build/platform/tmp/work/x86_64-linux/gdb-cross-aarch64/14.1/recipe-sysroot-native/usr/lib/libpython3.12.so.1.0 (0x00007fc765800000)
libmpfr.so.6 => /var/gbmc/gbmc/build/platform/tmp/work/x86_64-linux/gdb-cross-aarch64/14.1/recipe-sysroot-native/usr/lib/libmpfr.so.6 (0x00007fc76574d000)
libgmp.so.10 => /var/gbmc/gbmc/build/platform/tmp/work/x86_64-linux/gdb-cross-aarch64/14.1/recipe-sysroot-native/usr/lib/libgmp.so.10 (0x00007fc765e97000)
libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fc765400000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fc76566e000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fc765e6a000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc76521c000)
/lib64/ld-linux-x86-64.so.2 (0x00007fc766b0f000)
If you hit python issue, build GDB without python. One way is via PACKAGECONFIG at poky/meta/recipes-devtools/gdb/gdb-cross.inc
# remove python from the PACKAGECONFIG
PACKAGECONFIG ??= "readline ${@bb.utils.filter('DISTRO_FEATURES', 'debuginfod', d)}"Make sure GDB can be invoked.
/var/gbmc/coredump$ ./aarch64-openbmc-linux-gdb
GNU gdb (GDB) 14.1
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "--host=x86_64-linux --target=aarch64-openbmc-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
(gdb)
Compile daemon
Compile the target daemon and copy the unstripped binary to our workspace.
$ bitbake bmcweb
$ cp /var/gbmc/gbmc/build/platform/tmp/work/cortexa35-openbmc-linux/bmcweb/1.0+git/image/usr/bin/bmcweb /var/gbmc/coredump
/var/gbmc/coredump$ file bmcweb
bmcweb: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /usr/lib/ld-linux-aarch64.so.1, BuildID[sha1]=5730e9b505017d83e091b74f3bb639963208d936, for GNU/Linux 5.15.0, with debug_info, not stripped
Run GDB and resolve missing symbols
Now load the coredump into GDB.
[hi on] nanzhou@nanzhou-bmc:/var/gbmc/coredump$ ./aarch64-openbmc-linux-gdb bmcweb --core bmcweb.core
GNU gdb (GDB) 14.1
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "--host=x86_64-linux --target=aarch64-openbmc-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from bmcweb...
warning: Can't open file /usr/bin/bmcweb during file-backed mapping note processing
warning: Can't open file /usr/lib/libabsl_crc_internal.so.0 during file-backed mapping note processing
You will find it complains about missing files. One way to resolve it, is to copy every dependency from sysroot, including the binary itself, into the workspace, keeping the same relative path.
/var/gbmc/coredump$ mkdir -p usr/lib/
/var/gbmc/coredump$ mkdir -p usr/bin/
/var/gbmc/coredump$ mkdir -p lib
# Some weird libraries are needed to be in /lib. Here we just copy all libs to /lib and /usr/lib
$ cp /var/gbmc/gbmc/build/platform/tmp/work/cortexa35-openbmc-linux/bmcweb/1.0+git/recipe-sysroot/usr/lib/* /var/gbmc/coredump/usr/lib/ -r
$ cp /var/gbmc/gbmc/build/platform/tmp/work/cortexa35-openbmc-linux/bmcweb/1.0+git/recipe-sysroot/usr/lib/* /var/gbmc/coredump/lib/ -r
$ cp /var/gbmc/gbmc/build/platform/tmp/work/cortexa35-openbmc-linux/bmcweb/1.0+git/image/usr/bin/bmcweb /var/gbmc/coredump/usr/bin/
Verify all (or most) shared libraries are found.
(gdb) info sharedlibrary
From To Syms Read Shared Object Library
0x0000007fb11d2a90 0x0000007fb11da128 Yes (*) /var/gbmc/coredump/lib/libpam.so.0.85.1
0x0000007fb10f1e20 0x0000007fb1174480 Yes (*) /var/gbmc/coredump/lib/libssl.so.3
0x0000007fb0caf000 0x0000007fb0ef18b4 Yes (*) /var/gbmc/coredump/lib/libcrypto.so.3
0x0000007fb0bc8cc0 0x0000007fb0bd4f90 Yes (*) /var/gbmc/coredump/lib/libsdbusplus.so.1.0.0
0x0000007fb0ad94b0 0x0000007fb0b615e0 Yes (*) /var/gbmc/coredump/lib/libsystemd.so.0.38.0
0x0000007fb0a887b0 0x0000007fb0a91b8c Yes (*) /var/gbmc/coredump/lib/libtinyxml2.so.10.0.0
No /lib/libz.so.1
0x0000007fb09d5020 0x0000007fb0a04b40 Yes (*) /var/gbmc/coredump/lib/libboost_url.so.1.83.0
0x0000007fb0992070 0x0000007fb0992e60 Yes (*) /var/gbmc/coredump/lib/libboost_coroutine.so.1.83.0
0x0000007fb0960960 0x0000007fb0960d50 Yes (*) /var/gbmc/coredump/lib/libboost_context.so.1.83.0
0x0000007fb0918910 0x0000007fb092cc9c Yes (*) /var/gbmc/coredump/lib/libboost_filesystem.so.1.83.0
Now you should be able to see backtraces.
(gdb) thread apply all bt
Thread 23 (LWP 185495):
#0 syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
#1 0x0000007fb07e5be8 in absl::synchronization_internal::FutexWaiter::WaitUntil(std::atomic<int>*, int, absl::synchronization_internal::KernelTimeout) () from /var/gbmc/coredump/lib/libabsl_synchronization.so.0
#2 0x0000007fb07e5cb0 in absl::synchronization_internal::FutexWaiter::Wait(absl::synchronization_internal::KernelTimeout) () from /var/gbmc/coredump/lib/libabsl_synchronization.so.0
#3 0x0000007fb07e5f90 in AbslInternalPerThreadSemWait () from /var/gbmc/coredump/lib/libabsl_synchronization.so.0
#4 0x0000007fb07ea1ac in absl::CondVar::WaitCommon(absl::Mutex*, absl::synchronization_internal::KernelTimeout) () from /var/gbmc/coredump/lib/libabsl_synchronization.so.0
#5 0x0000007fb026f538 in ?? ()
#6 0x0000007f897ae330 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)