Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[SOLVED] How to request a backtrace?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo
View previous topic :: View next topic  
Author Message
neyuru
Apprentice
Apprentice


Joined: 21 Mar 2020
Posts: 177

PostPosted: Tue Nov 17, 2020 1:16 am    Post subject: [SOLVED] How to request a backtrace? Reply with quote

I recently filed a bug in gentoo bugzilla, but the maintainers need more information from me. A backtrace was suggested to pin down the problem. Unfortunately, I don't know how to do it.

The setup:
A program that initializes automatically at boot. I know (from another post) that strace could be used but I only know how to use it if I start the program. What about a program that gets started at boot? For reference, I am using systemd.

thanks!


Last edited by neyuru on Tue Nov 17, 2020 2:47 pm; edited 1 time in total
Back to top
View user's profile Send private message
Etal
Veteran
Veteran


Joined: 15 Jul 2005
Posts: 1865

PostPosted: Tue Nov 17, 2020 2:08 am    Post subject: Reply with quote

You can find a bunch of information about generating meaningful backtraces here: https://wiki.gentoo.org/wiki/Project:Quality_Assurance/Backtraces

If you're looking for something more concrete, you'll need to provide more specific info... Is it crashing or misbehaving some other way? Can you start it from command line not during boot?
Back to top
View user's profile Send private message
neyuru
Apprentice
Apprentice


Joined: 21 Mar 2020
Posts: 177

PostPosted: Tue Nov 17, 2020 2:24 am    Post subject: Reply with quote

Etal wrote:
You can find a bunch of information about generating meaningful backtraces here: https://wiki.gentoo.org/wiki/Project:Quality_Assurance/Backtraces

If you're looking for something more concrete, you'll need to provide more specific info... Is it crashing or misbehaving some other way? Can you start it from command line not during boot?


Thanks for the info! I will look into it. About the bug, I'm only able to tell that its related to the xfce4 settings daemon xfcesettingsd: (xfce-base/xfce4-settings) which is automatically started at boot e.g. it's not a service I manually placed in the stack. I have given a full log of events at the reported bug. Maybe there's a way to call it manually? :?:
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 16456

PostPosted: Tue Nov 17, 2020 2:40 am    Post subject: Reply with quote

If the process dumps core when started at boot, you can open the core file in gdb and get the backtrace that way.
Back to top
View user's profile Send private message
neyuru
Apprentice
Apprentice


Joined: 21 Mar 2020
Posts: 177

PostPosted: Tue Nov 17, 2020 4:15 am    Post subject: Reply with quote

Thank for the help! One question though: where is the core dump file?
Back to top
View user's profile Send private message
neyuru
Apprentice
Apprentice


Joined: 21 Mar 2020
Posts: 177

PostPosted: Tue Nov 17, 2020 4:19 am    Post subject: Reply with quote

Sorry I think I found it... one problem though. I ran:

Code:
gdb -q xfsettingsd --core /var/lib/systemd/coredump/core.xfsettingsd.1000.995002ad376d4526ad4d4405fd21e134.2792.1605403663000000.zst


because it's supposed systemd dumps in that directory. But gdb gives me this error:

Code:
Reading symbols from xfsettingsd...
"/var/lib/systemd/coredump/core.xfsettingsd.1000.995002ad376d4526ad4d4405fd21e134.2792.1605403663000000.zst" is not a core dump: file format not recognized
Back to top
View user's profile Send private message
neyuru
Apprentice
Apprentice


Joined: 21 Mar 2020
Posts: 177

PostPosted: Tue Nov 17, 2020 4:58 am    Post subject: Reply with quote

I used systemd's

Code:
coredumpctl debug


to view the last generated dump (supposedly). This is what it gave:

Code:
coredumpctl debug
           PID: 3804 (xfsettingsd)
           UID: 1000 (jorge)
           GID: 1000 (jorge)
        Signal: 5 (TRAP)
     Timestamp: Mon 2020-11-16 22:41:12 CST (12min ago)
  Command Line: xfsettingsd --display :0.0 --sm-client-id 23526dd8b-908a-4cdb-8e1d-cb3ea2a7c698
    Executable: /usr/bin/xfsettingsd
 Control Group: /user.slice/user-1000.slice/session-c2.scope
          Unit: session-c2.scope
         Slice: user-1000.slice
       Session: c2
     Owner UID: 1000 (jorge)
       Boot ID: 03f5150313b242a092c15a8af23a77b9
    Machine ID: 170173c64e6b37a0c863c41e5ee1a2ee
      Hostname: localhost
       Storage: /var/lib/systemd/coredump/core.xfsettingsd.1000.03f5150313b242a092c15a8af23a77b9.3804.1605588072000000.zst
       Message: Process 3804 (xfsettingsd) of user 1000 dumped core.

GNU gdb (Gentoo 9.2 vanilla) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/xfsettingsd...
[New LWP 3804]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `xfsettingsd --display :0.0 --sm-client-id 23526dd8b-908a-4cdb-8e1d-cb3ea2a7c698'.
Program terminated with signal SIGTRAP, Trace/breakpoint trap.
#0  0x00007f2c768a4627 in g_log_structured_array () from /usr/lib64/libglib-2.0.so.0


is this information usefull?
Back to top
View user's profile Send private message
Etal
Veteran
Veteran


Joined: 15 Jul 2005
Posts: 1865

PostPosted: Tue Nov 17, 2020 12:06 pm    Post subject: Reply with quote

When you do "coredumpctl debug" it should leave you with a "(gdb)" prompt. Run "bt" to get a backtrace.
Back to top
View user's profile Send private message
neyuru
Apprentice
Apprentice


Joined: 21 Mar 2020
Posts: 177

PostPosted: Tue Nov 17, 2020 2:47 pm    Post subject: Reply with quote

Thank you all for the helpful info!
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 16456

PostPosted: Tue Nov 17, 2020 4:49 pm    Post subject: Reply with quote

neyuru wrote:
Thank for the help! One question though: where is the core dump file?
Traditionally, it is written to a file named core in the current working directory of the process that died. Based on your later output, systemd has deviated from this tradition. You will need to use systemd tools (as you discovered on your own) to find the file.
neyuru wrote:
Code:
gdb -q xfsettingsd --core /var/lib/systemd/coredump/core.xfsettingsd.1000.995002ad376d4526ad4d4405fd21e134.2792.1605403663000000.zst
Something, probably related to the systemd logic that redirected the file here, compressed the core dump file using zstd. You will need an uncompressed copy in order to proceed. I see in later posts that you managed to get the information you wanted, but I thought I should make this post to explain why you needed to take the path that you did.
neyuru wrote:
Code:
Core was generated by `xfsettingsd --display :0.0 --sm-client-id 23526dd8b-908a-4cdb-8e1d-cb3ea2a7c698'.
Program terminated with signal SIGTRAP, Trace/breakpoint trap.
#0  0x00007f2c768a4627 in g_log_structured_array () from /usr/lib64/libglib-2.0.so.0
is this information usefull?
On its own, it is slightly helpful. It confirms what we knew from the kernel's log messages: the program terminated itself, probably in reaction to a failed assertion of some sort. The backtrace will tell how the program reached the function that terminated it, which is more useful in understanding why it decided to terminate.
Back to top
View user's profile Send private message
neyuru
Apprentice
Apprentice


Joined: 21 Mar 2020
Posts: 177

PostPosted: Wed Nov 18, 2020 3:09 am    Post subject: Reply with quote

Hu wrote:
I see in later posts that you managed to get the information you wanted, but I thought I should make this post to explain why you needed to take the path that you did.


Great idea and great info! In the case someone sees this post and is using systemd, all Hu has said is correct. The systemd tool named coredumpctl works along gdb. gdb is the default debugger used in coredumpctl but, of course, you have to install it by itself. In the example given above, I may add that further debugging is possible, just like when working with gdb, you just have to issue a "bt" command in the gdb prompt, like so (look at the seventh to last line):

Code:
coredumpctl debug xfsettingsd
           PID: 4788 (xfsettingsd)
           UID: 1000 (jorge)
           GID: 1000 (jorge)
        Signal: 5 (TRAP)
     Timestamp: Tue 2020-11-17 19:14:00 CST (1h 49min ago)
  Command Line: xfsettingsd --display :0.0 --sm-client-id 269c59d9f-4e3b-447b-b75e-020ff9963578
    Executable: /usr/bin/xfsettingsd
 Control Group: /user.slice/user-1000.slice/session-c2.scope
          Unit: session-c2.scope
         Slice: user-1000.slice
       Session: c2
     Owner UID: 1000 (jorge)
       Boot ID: e5e0d11dfd904943991708dee6d931d8
    Machine ID: 170173c64e6b37a0c863c41e5ee1a2ee
      Hostname: localhost
       Storage: /var/lib/systemd/coredump/core.xfsettingsd.1000.e5e0d11dfd904943991708dee6d931d8.4788.1605662040000000.zst
       Message: Process 4788 (xfsettingsd) of user 1000 dumped core.

GNU gdb (Gentoo 9.2 vanilla) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/xfsettingsd...
[New LWP 4788]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `xfsettingsd --display :0.0 --sm-client-id 269c59d9f-4e3b-447b-b75e-020ff9963578'.
Program terminated with signal SIGTRAP, Trace/breakpoint trap.
#0  0x00007f4a1ea2f627 in g_log_structured_array () from /usr/lib64/libglib-2.0.so.0
(gdb) bt
#0  0x00007f4a1ea2f627 in g_log_structured_array () at /usr/lib64/libglib-2.0.so.0
#1  0x00007f4a1ea2fabc in g_log_default_handler () at /usr/lib64/libglib-2.0.so.0
#2  0x00007f4a1ea2fd49 in g_logv () at /usr/lib64/libglib-2.0.so.0
#3  0x00007f4a1ea2ffc7 in g_log () at /usr/lib64/libglib-2.0.so.0
#4  0x0000562f6c50b8aa in main (argc=<optimized out>, argv=<optimized out>) at main.c:281
(gdb) quit


Systemd will compress the coredumps in those files ending with .zst and coredumpctl can work on them or it can uncompress them if you want to with something like

Code:
coredumpctl -o bar.coredump dump /usr/bin/bar


(taken from the man page). One thing I don't like about this tool though is that it seems only can handle the last coredump generated and I don't seem to find an easy way to debug previous ocurrences.
Back to top
View user's profile Send private message
Etal
Veteran
Veteran


Joined: 15 Jul 2005
Posts: 1865

PostPosted: Thu Nov 19, 2020 11:04 pm    Post subject: Reply with quote

You can run "coredumpctl" to see the list of core dumps, and then select the one you want by PID.

Code:
# coredumpctl
TIME                            PID   UID   GID SIG COREFILE  EXE
Tue 2020-11-17 14:13:13 EST  2902068   250   250   6 error     /var/tmp/portage/sys-devel/bison-3.7.4/work/bison-3.7.4/conftest
Tue 2020-11-17 14:13:21 EST  2904277   250   250   6 error     /var/tmp/portage/sys-devel/bison-3.7.4/work/bison-3.7.4/conftest
Tue 2020-11-17 15:46:26 EST  3201039   250   250   6 error     /var/tmp/portage/app-portage/portage-utils-0.90.1/work/portage-utils-0.90.1/conftest


So for if you wanted the first one, you can run:
Code:
# coredumpctl debug 2902068
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum