Let’s say a daemon got a segmentation fault.<\/p>\n\n\n\n
$ coredumpctl\n\nTIME PID UID GID SIG COREFILE EXE SIZE\nTue 9999-99-99 00:00:00 PDT 12345 0 0 SIGSEGV present \/usr\/bin\/bmcweb 20.7M<\/code><\/pre>\n\n\n\nDump the core and SCP it form the machine to your workstation.<\/p>\n\n\n\n
$ coredumpctl dump 12345 > \/tmp\/bmcweb.core<\/code><\/pre>\n\n\n\nPrepare new workspace<\/h2>\n\n\n\n
Create a brand new workspace and copy require files over.<\/p>\n\n\n\n
$ mkdir -p \/var\/gbmc\/coredump<\/code><\/pre>\n\n\n\nCompile GDB<\/h2>\n\n\n\n
Depending on your target machine, build the corresponding cross GDB.<\/p>\n\n\n\n
$ bitbake gdb-cross-aarch64\n# bitbake gdb-cross-arm for 32 bits machine\n\n$ cp \/var\/gbmc\/gbmc\/build\/platform\/tmp\/work\/x86_64-linux\/gdb-cross-aarch64\/14.1\/image\/var\/gbmc\/gbmc\/build\/platform\/tmp\/work\/x86_64-linux\/gdb-cross-aarch64\/14.1\/recipe-sysroot-native\/usr\/bin\/aarch64-openbmc-linux\/aarch64-openbmc-linux-gdb \/var\/gbmc\/coredump\/\n\n\/var\/gbmc\/coredump$ ldd aarch64-openbmc-linux-gdb \n\tlinux-vdso.so.1 (0x00007fc766b0d000)\n\tlibexpat.so.1 => \/var\/gbmc\/gbmc\/build\/platform\/tmp\/work\/x86_64-linux\/gdb-cross-aarch64\/14.1\/recipe-sysroot-native\/usr\/lib\/libexpat.so.1 (0x00007fc766add000)\n\tlibreadline.so.8 => \/var\/gbmc\/gbmc\/build\/platform\/tmp\/work\/x86_64-linux\/gdb-cross-aarch64\/14.1\/recipe-sysroot-native\/usr\/lib\/libreadline.so.8 (0x00007fc766a86000)\n\tlibz.so.1 => \/var\/gbmc\/gbmc\/build\/platform\/tmp\/work\/x86_64-linux\/gdb-cross-aarch64\/14.1\/recipe-sysroot-native\/usr\/lib\/libz.so.1 (0x00007fc766a6c000)\n\tlibzstd.so.1 => \/var\/gbmc\/gbmc\/build\/platform\/tmp\/work\/x86_64-linux\/gdb-cross-aarch64\/14.1\/recipe-sysroot-native\/usr\/lib\/libzstd.so.1 (0x00007fc765f3e000)\n\tlibtinfo.so.5 => \/var\/gbmc\/gbmc\/build\/platform\/tmp\/work\/x86_64-linux\/gdb-cross-aarch64\/14.1\/recipe-sysroot-native\/usr\/lib\/libtinfo.so.5 (0x00007fc765f0f000)\n\tlibpython3.12.so.1.0 => \/var\/gbmc\/gbmc\/build\/platform\/tmp\/work\/x86_64-linux\/gdb-cross-aarch64\/14.1\/recipe-sysroot-native\/usr\/lib\/libpython3.12.so.1.0 (0x00007fc765800000)\n\tlibmpfr.so.6 => \/var\/gbmc\/gbmc\/build\/platform\/tmp\/work\/x86_64-linux\/gdb-cross-aarch64\/14.1\/recipe-sysroot-native\/usr\/lib\/libmpfr.so.6 (0x00007fc76574d000)\n\tlibgmp.so.10 => \/var\/gbmc\/gbmc\/build\/platform\/tmp\/work\/x86_64-linux\/gdb-cross-aarch64\/14.1\/recipe-sysroot-native\/usr\/lib\/libgmp.so.10 (0x00007fc765e97000)\n\tlibstdc++.so.6 => \/lib\/x86_64-linux-gnu\/libstdc++.so.6 (0x00007fc765400000)\n\tlibm.so.6 => \/lib\/x86_64-linux-gnu\/libm.so.6 (0x00007fc76566e000)\n\tlibgcc_s.so.1 => \/lib\/x86_64-linux-gnu\/libgcc_s.so.1 (0x00007fc765e6a000)\n\tlibc.so.6 => \/lib\/x86_64-linux-gnu\/libc.so.6 (0x00007fc76521c000)\n\t\/lib64\/ld-linux-x86-64.so.2 (0x00007fc766b0f000)\n<\/code><\/pre>\n\n\n\nIf you hit python issue, build GDB without python. One way is via PACKAGECONFIG at poky\/meta\/recipes-devtools\/gdb\/gdb-cross.inc<\/code><\/p>\n\n\n\n# remove python from the PACKAGECONFIG\nPACKAGECONFIG ??= \"readline ${@bb.utils.filter('DISTRO_FEATURES', 'debuginfod', d)}\"<\/code><\/pre>\n\n\n\nMake sure GDB can be invoked.<\/p>\n\n\n\n
\/var\/gbmc\/coredump$ .\/aarch64-openbmc-linux-gdb\nGNU gdb (GDB) 14.1\nCopyright (C) 2023 Free Software Foundation, Inc.\nLicense GPLv3+: GNU GPL version 3 or later <http:\/\/gnu.org\/licenses\/gpl.html>\nThis is free software: you are free to change and redistribute it.\nThere is NO WARRANTY, to the extent permitted by law.\nType \"show copying\" and \"show warranty\" for details.\nThis GDB was configured as \"--host=x86_64-linux --target=aarch64-openbmc-linux\".\nType \"show configuration\" for configuration details.\nFor bug reporting instructions, please see:\n<https:\/\/www.gnu.org\/software\/gdb\/bugs\/>.\nFind the GDB manual and other documentation resources online at:\n <http:\/\/www.gnu.org\/software\/gdb\/documentation\/>.\n\nFor help, type \"help\".\nType \"apropos word\" to search for commands related to \"word\".\n(gdb) \n<\/code><\/pre>\n\n\n\nCompile daemon<\/h2>\n\n\n\n
Compile the target daemon and copy the unstripped binary to our workspace.<\/p>\n\n\n\n
$ bitbake bmcweb\n$ cp \/var\/gbmc\/gbmc\/build\/platform\/tmp\/work\/cortexa35-openbmc-linux\/bmcweb\/1.0+git\/image\/usr\/bin\/bmcweb \/var\/gbmc\/coredump\n\n\/var\/gbmc\/coredump$ file bmcweb\nbmcweb: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter \/usr\/lib\/ld-linux-aarch64.so.1, BuildID[sha1]=5730e9b505017d83e091b74f3bb639963208d936, for GNU\/Linux 5.15.0, with debug_info, not stripped\n<\/code><\/pre>\n\n\n\nRun GDB and resolve missing symbols<\/h2>\n\n\n\n
Now load the coredump into GDB.<\/p>\n\n\n\n
[hi on] nanzhou@nanzhou-bmc:\/var\/gbmc\/coredump$ .\/aarch64-openbmc-linux-gdb bmcweb --core bmcweb.core \nGNU gdb (GDB) 14.1\nCopyright (C) 2023 Free Software Foundation, Inc.\nLicense GPLv3+: GNU GPL version 3 or later <http:\/\/gnu.org\/licenses\/gpl.html>\nThis is free software: you are free to change and redistribute it.\nThere is NO WARRANTY, to the extent permitted by law.\nType \"show copying\" and \"show warranty\" for details.\nThis GDB was configured as \"--host=x86_64-linux --target=aarch64-openbmc-linux\".\nType \"show configuration\" for configuration details.\nFor bug reporting instructions, please see:\n<https:\/\/www.gnu.org\/software\/gdb\/bugs\/>.\nFind the GDB manual and other documentation resources online at:\n <http:\/\/www.gnu.org\/software\/gdb\/documentation\/>.\n\nFor help, type \"help\".\nType \"apropos word\" to search for commands related to \"word\"...\nReading symbols from bmcweb...\n\nwarning: Can't open file \/usr\/bin\/bmcweb during file-backed mapping note processing\n\nwarning: Can't open file \/usr\/lib\/libabsl_crc_internal.so.0 during file-backed mapping note processing\n\n<\/code><\/pre>\n\n\n\nYou will find it complains about missing files. One way to resolve it, is to copy every dependency from sysroot, including the binary itself, into the workspace, keeping the same relative path.<\/p>\n\n\n\n
\/var\/gbmc\/coredump$ mkdir -p usr\/lib\/\n\/var\/gbmc\/coredump$ mkdir -p usr\/bin\/\n\/var\/gbmc\/coredump$ mkdir -p lib\n\n# Some weird libraries are needed to be in \/lib. Here we just copy all libs to \/lib and \/usr\/lib\n$ cp \/var\/gbmc\/gbmc\/build\/platform\/tmp\/work\/cortexa35-openbmc-linux\/bmcweb\/1.0+git\/recipe-sysroot\/usr\/lib\/* \/var\/gbmc\/coredump\/usr\/lib\/ -r\n$ cp \/var\/gbmc\/gbmc\/build\/platform\/tmp\/work\/cortexa35-openbmc-linux\/bmcweb\/1.0+git\/recipe-sysroot\/usr\/lib\/* \/var\/gbmc\/coredump\/lib\/ -r\n$ cp \/var\/gbmc\/gbmc\/build\/platform\/tmp\/work\/cortexa35-openbmc-linux\/bmcweb\/1.0+git\/image\/usr\/bin\/bmcweb \/var\/gbmc\/coredump\/usr\/bin\/\n<\/code><\/pre>\n\n\n\nVerify all (or most) shared libraries are found.<\/p>\n\n\n\n
(gdb) info sharedlibrary \nFrom To Syms Read Shared Object Library\n0x0000007fb11d2a90 0x0000007fb11da128 Yes (*) \/var\/gbmc\/coredump\/lib\/libpam.so.0.85.1\n0x0000007fb10f1e20 0x0000007fb1174480 Yes (*) \/var\/gbmc\/coredump\/lib\/libssl.so.3\n0x0000007fb0caf000 0x0000007fb0ef18b4 Yes (*) \/var\/gbmc\/coredump\/lib\/libcrypto.so.3\n0x0000007fb0bc8cc0 0x0000007fb0bd4f90 Yes (*) \/var\/gbmc\/coredump\/lib\/libsdbusplus.so.1.0.0\n0x0000007fb0ad94b0 0x0000007fb0b615e0 Yes (*) \/var\/gbmc\/coredump\/lib\/libsystemd.so.0.38.0\n0x0000007fb0a887b0 0x0000007fb0a91b8c Yes (*) \/var\/gbmc\/coredump\/lib\/libtinyxml2.so.10.0.0\n No \/lib\/libz.so.1\n0x0000007fb09d5020 0x0000007fb0a04b40 Yes (*) \/var\/gbmc\/coredump\/lib\/libboost_url.so.1.83.0\n0x0000007fb0992070 0x0000007fb0992e60 Yes (*) \/var\/gbmc\/coredump\/lib\/libboost_coroutine.so.1.83.0\n0x0000007fb0960960 0x0000007fb0960d50 Yes (*) \/var\/gbmc\/coredump\/lib\/libboost_context.so.1.83.0\n0x0000007fb0918910 0x0000007fb092cc9c Yes (*) \/var\/gbmc\/coredump\/lib\/libboost_filesystem.so.1.83.0\n<\/code><\/pre>\n\n\n\nNow you should be able to see backtraces.<\/p>\n\n\n\n
(gdb) thread apply all bt\n\nThread 23 (LWP 185495):\n#0 syscall () at ..\/sysdeps\/unix\/sysv\/linux\/aarch64\/syscall.S:38\n#1 0x0000007fb07e5be8 in absl::synchronization_internal::FutexWaiter::WaitUntil(std::atomic<int>*, int, absl::synchronization_internal::KernelTimeout) () from \/var\/gbmc\/coredump\/lib\/libabsl_synchronization.so.0\n#2 0x0000007fb07e5cb0 in absl::synchronization_internal::FutexWaiter::Wait(absl::synchronization_internal::KernelTimeout) () from \/var\/gbmc\/coredump\/lib\/libabsl_synchronization.so.0\n#3 0x0000007fb07e5f90 in AbslInternalPerThreadSemWait () from \/var\/gbmc\/coredump\/lib\/libabsl_synchronization.so.0\n#4 0x0000007fb07ea1ac in absl::CondVar::WaitCommon(absl::Mutex*, absl::synchronization_internal::KernelTimeout) () from \/var\/gbmc\/coredump\/lib\/libabsl_synchronization.so.0\n#5 0x0000007fb026f538 in ?? ()\n#6 0x0000007f897ae330 in ?? ()\nBacktrace stopped: previous frame inner to this frame (corrupt stack?)\n<\/code><\/pre>\n\n\n\n<\/p>\n","protected":false},"excerpt":{"rendered":"
Got a coredump Let’s say a daemon got a segmentation fault. Dump the core and SCP it form the machine to your workstation. Prepare new workspace Create a brand new workspace and copy require files over. Compile GDB Depending on your target machine, build the corresponding cross GDB. If you hit python issue, build GDB…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"0","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[73,72],"tags":[],"class_list":["post-1746","post","type-post","status-publish","format-standard","hentry","category-debug","category-openbmc"],"_links":{"self":[{"href":"https:\/\/nanzhou.cc\/index.php\/wp-json\/wp\/v2\/posts\/1746","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nanzhou.cc\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nanzhou.cc\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nanzhou.cc\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nanzhou.cc\/index.php\/wp-json\/wp\/v2\/comments?post=1746"}],"version-history":[{"count":2,"href":"https:\/\/nanzhou.cc\/index.php\/wp-json\/wp\/v2\/posts\/1746\/revisions"}],"predecessor-version":[{"id":1749,"href":"https:\/\/nanzhou.cc\/index.php\/wp-json\/wp\/v2\/posts\/1746\/revisions\/1749"}],"wp:attachment":[{"href":"https:\/\/nanzhou.cc\/index.php\/wp-json\/wp\/v2\/media?parent=1746"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nanzhou.cc\/index.php\/wp-json\/wp\/v2\/categories?post=1746"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nanzhou.cc\/index.php\/wp-json\/wp\/v2\/tags?post=1746"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}