| View previous topic :: View next topic |
| Author |
Message |
Havin_it
Veteran
Joined: 17 Jul 2005
Posts: 1247
Location: Edinburgh, UK
|
 Posted: Wed Nov 05, 2008 5:24 am Post subject: Squeezecenter: memory corruption |
 |
|
Hi,
With media-sound/squeezecenter-7.2.1, the last time I rebooted my server (after many updates over a period of a month or so) squeezecenter failed to start. When I try to launch it from the shell, I get this:
| Code: | brazil htdocs # /etc/init.d/squeezecenter start
* Starting SqueezeCenter ...
*** glibc detected *** start-stop-daemon: malloc(): memory corruption: 0x09b36160 ***
======= Backtrace: =========
/lib/libc.so.6[0xb7f6a3b2]
/lib/libc.so.6[0xb7f6cbfa]
/lib/libc.so.6(__libc_malloc+0x9e)[0xb7f6e86e]
/lib/libc.so.6[0xb7f2a681]
/lib/libc.so.6(setenv+0x71)[0xb7f2a971]
start-stop-daemon[0x804f6f1]
start-stop-daemon[0x80509c1]
start-stop-daemon[0x805930f]
/lib/libc.so.6(__libc_start_main+0xe1)[0xb7f12621]
start-stop-daemon[0x804b8e1]
======= Memory map: ========
08048000-08060000 r-xp 00000000 08:05 883027 /sbin/rc
08060000-08061000 r--p 00017000 08:05 883027 /sbin/rc
08061000-08062000 rw-p 00018000 08:05 883027 /sbin/rc
08062000-08064000 rw-p 08062000 00:00 0
09b33000-09b54000 rw-p 09b33000 00:00 0 [heap]
b7d00000-b7d21000 rw-p b7d00000 00:00 0
b7d21000-b7e00000 ---p b7d21000 00:00 0
b7e9f000-b7eab000 r-xp 00000000 08:05 1898252 /lib/libgcc_s.so.1
b7eab000-b7eac000 r--p 0000b000 08:05 1898252 /lib/libgcc_s.so.1
b7eac000-b7ead000 rw-p 0000c000 08:05 1898252 /lib/libgcc_s.so.1
b7ead000-b7eb6000 r-xp 00000000 08:05 1684286 /lib/libnss_files-2.8.so
b7eb6000-b7eb7000 r--p 00008000 08:05 1684286 /lib/libnss_files-2.8.so
b7eb7000-b7eb8000 rw-p 00009000 08:05 1684286 /lib/libnss_files-2.8.so
b7ec3000-b7ed7000 r-xp 00000000 08:05 1684360 /lib/libnsl-2.8.so
b7ed7000-b7ed8000 r--p 00013000 08:05 1684360 /lib/libnsl-2.8.so
b7ed8000-b7ed9000 rw-p 00014000 08:05 1684360 /lib/libnsl-2.8.so
b7ed9000-b7edb000 rw-p b7ed9000 00:00 0
b7edc000-b7ee4000 r-xp 00000000 08:05 1684668 /lib/libnss_nis-2.8.so
b7ee4000-b7ee5000 r--p 00008000 08:05 1684668 /lib/libnss_nis-2.8.so
b7ee5000-b7ee6000 rw-p 00009000 08:05 1684668 /lib/libnss_nis-2.8.so
b7ee6000-b7eed000 r-xp 00000000 08:05 1684372 /lib/libnss_compat-2.8.so
b7eed000-b7eee000 r--p 00006000 08:05 1684372 /lib/libnss_compat-2.8.so
b7eee000-b7eef000 rw-p 00007000 08:05 1684372 /lib/libnss_compat-2.8.so
b7eef000-b7ef2000 r-xp 00000000 08:05 1685128 /lib/security/pam_limits.so
b7ef2000-b7ef3000 r--p 00002000 08:05 1685128 /lib/security/pam_limits.so
b7ef3000-b7ef4000 rw-p 00003000 08:05 1685128 /lib/security/pam_limits.so
b7ef4000-b7ef5000 r-xp 00000000 08:05 1685118 /lib/security/pam_deny.so
b7ef5000-b7ef6000 r--p 00000000 08:05 1685118 /lib/security/pam_deny.so
b7ef6000-b7ef7000 rw-p 00001000 08:05 1685118 /lib/security/pam_deny.so
b7ef7000-b7ef8000 r-xp 00000000 08:05 1685135 /lib/security/pam_permit.so
b7ef8000-b7ef9000 r--p 00000000 08:05 1685135 /lib/security/pam_permit.so
b7ef9000-b7efa000 rw-p 00001000 08:05 1685135 /lib/security/pam_permit.so
b7efb000-b7efc000 rw-p b7efb000 00:00 0
b7efc000-b803f000 r-xp 00000000 08:05 1684682 /lib/libc-2.8.so
b803f000-b8041000 r--p 00143000 08:05 1684682 /lib/libc-2.8.so
b8041000-b8042000 rw-p 00145000 08:05 1684682 /lib/libc-2.8.so
b8042000-b8046000 rw-p b8042000 00:00 0
b8046000-b8050000 r-xp 00000000 08:05 1685105 /lib/libpam.so.0.81.12
b8050000-b8051000 r--p 00009000 08:05 1685105 /lib/libpam.so.0.81.12
b8051000-b8052000 rw-p 0000a000 08:05 1685105 /lib/libpam.so.0.81.12
b8052000-b8054000 r-xp 00000000 08:05 1684663 /lib/libdl-2.8.so
b8054000-b8055000 r--p 00001000 08:05 1684663 /lib/libdl-2.8.so
b8055000-b8056000 rw-p 00002000 08:05 1684663 /lib/libdl-2.8.so
b8056000-b8094000 r-xp 00000000 08:05 1684529 /lib/libncurses.so.5.6
b8094000-b809c000 r--p 0003d000 08:05 1684529 /lib/libncurses.so.5.6
b809c000-b809d000 rw-p 00045000 08:05 1684529 /lib/libncurses.so.5.6
b809d000-b809e000 rw-p b809d000 00:00 0
b809e000-b80a1000 r-xp 00000000 08:05 2305705 /lib/libeinfo.so.1
b80a1000-b80a2000 r--p 00003000 08:05 2305705 /lib/libeinfo.so.1
b80a2000-b80a3000 rw-p 00004000 08:05 2305705 /lib/libeinfo.so.1
b80a3000-b80ac000 r-xp 00000000 08:05 2305703 /lib/librc.so.1
b80ac000-b80ad000 r--p 00008000 08:05 2305703 /lib/librc.so.1
b80ad000-b80ae000 rw-p 00009000 08:05 2305703 /lib/librc.so.1
b80ae000-b80b0000 r-xp 00000000 08:05 1684670 /lib/libutil-2.8.so
b80b0000-b80b1000 r--p 00001000 08:05 1684670 /lib/libutil-2.8.so
b80b1000-b80b2000 rw-p 00002000 08:05 1684670 /lib/libutil-2.8.so
b80b2000-b80b3000 rw-p b80b2000 00:00 0
b80b3000-b80cf000 r-xp 00000000 08:05 1684681 /lib/ld-2.8.so
b80cf000-b80d0000 r--p 0001b000 08:05 1684681 /lib/ld-2.8.so
b80d0000-b80d1000 rw-p 0001c000 08:05 1684681 /lib/ld-2.8.so
bfabc000-bfad1000 rw-p bffeb000 00:00 0 [stack]
ffffe000-fffff000 r-xp 00000000 00:00 0 [vdso]
* start-stop-daemon: failed to start `/usr/bin/perl'
* Failed to start SqueezeCenter [ !! ]
* ERROR: squeezecenter failed to start
|
I've done an emerge -uD world, revdep-rebuild (nothing to rebuild), @preserved-rebuild (rebuilt ffmpeg, seems to do that a lot) and remerged perl, openrc and squeezecenter, without result.
What's a boy to do? |
|
| Back to top |
|
 |
Hu
Moderator
Joined: 06 Mar 2007
Posts: 21639
|
| Posted: Thu Nov 06, 2008 4:00 am Post subject: |
|
|
| Build libc and squeezecenter with debug symbols, and track down the corruption. See How to get meaningful backtraces in Gentoo for instructions on how to build symbols. Report the corruption as a bug. If squeezecenter has any Gentoo patches that could be contributing to it, start with a Gentoo bug, but be prepared to take the issue upstream. If Gentoo is not patching squeezecenter, go directly to upstream. When upstream issues a fix, file a Gentoo bug to have the fix incorporated into the tree. |
|
| Back to top |
|
|
Havin_it
Veteran
Joined: 17 Jul 2005
Posts: 1247
Location: Edinburgh, UK
|
| Posted: Thu Nov 06, 2008 3:09 pm Post subject: |
|
|
Hi, thanks for the info. Squeezecenter is all Perl, so I'm rebuilding glibc and openrc for debug. Should I do the same with perl?
Also, do I need to reboot after rebuilding before I can get a proper backtrace? |
|
| Back to top |
|
|
Havin_it
Veteran
Joined: 17 Jul 2005
Posts: 1247
Location: Edinburgh, UK
|
| Posted: Fri Nov 07, 2008 1:16 pm Post subject: |
|
|
I fear I've either done something wrong or there's something else I still need to do. I rebuilt glibc and openrc with the following command:
| Code: | | FEATURES="nostrip" USE="debug" CFLAGS="${CFLAGS} -g" CXXFLAGS="${CFLAGS}" emerge -1 glibc openrc |
I also rebooted, but when I tried to start squeezecenter, the output of the 'backtrace' was the same as before. Is this a proper backtrace? I realise I should probably be using gdb somehow, but it's such a complicated chain of execution (initscript -> /sbin/runscript -> start-stop-daemon -> /usr/sbin/squeezecenter-server -> perl) here that I'm not sure how to begin.
Can you instruct further?
PS - tried downgrading to squeezecenter-7.2.0-r1 (which definitely did work earlier) but the crash still occurred. It certainly seems the bug is not in squeezecenter itself, but in glibc or one of the other binaries involved. |
|
| Back to top |
|
|
Hu
Moderator
Joined: 06 Mar 2007
Posts: 21639
|
| Posted: Sat Nov 08, 2008 4:03 am Post subject: |
|
|
The crash appears to be in the initscript logic. It is not clear to me whether the problem is /sbin/rc or /sbin/start-stop-daemon. The error message implicates start-stop-daemon, but the file mappings implicate rc. Either way, I think it is failing before it has a chance to try to run perl. The easiest way to debug this might be to ensure that core file size is not limited, then set /proc/sys/kernel/core_pattern to a path where the dying process can write. Reproduce the crash, and a core file should be left in the location specified by core_pattern. You can then use gdb to analyze the failure.
A reboot was not required, but did not hurt. |
|
| Back to top |
|
|
Havin_it
Veteran
Joined: 17 Jul 2005
Posts: 1247
Location: Edinburgh, UK
|
| Posted: Sat Nov 08, 2008 1:05 pm Post subject: |
|
|
Well, I can clear up the issue about start-stop-daemon and rc: one is a symlink to the other 
Please can you give more detail on how to perform the dump operation you described above? Specifically: how do I ensure that core file size is not limited; and which process ( = which user) must be able to write to the location specified in /proc/sys/kernel/core_pattern? I tried two locations, /root (root issued the command) and /var/lib/squeezecenter (owned by user squeezecenter who the server should run as) but no file was created. Also, should it be a directory path or a file path (I tried both)? |
|
| Back to top |
|
|
Hu
Moderator
Joined: 06 Mar 2007
Posts: 21639
|
| Posted: Sat Nov 08, 2008 7:00 pm Post subject: |
|
|
I see. In =sys-apps/baselayout-1*, the two were not links to each other. I did not check a baselayout-2 system before posting. 
The core file size limit can be queried and set with ulimit -c. See info bash for details of how to change it, but if running ulimit -c prints unlimited, then you do not need to do anything here.
The value in core_pattern should refer to a file which does not exist yet, in a directory which can be written by the effective user id of the faulting process. In your case, the process is almost certainly root, so anywhere that root can write should be fine. You can also include some percent-escapes to encode information about the faulting process into the filename. See /usr/src/linux/fs/exec.c for the gory details, but the summary is:
- % - literal %
- p - pid
- u - uid
- g - gid
- s - number of fatal signal
- t - time of dump, as seconds since epoch
- h - hostname
- e - executable name
- c - core size limit
|
|
| Back to top |
|
|
Havin_it
Veteran
Joined: 17 Jul 2005
Posts: 1247
Location: Edinburgh, UK
|
| Posted: Sun Nov 09, 2008 11:54 am Post subject: |
|
|
| I set ulimit to 100,000,000 [sans commas] -- it had output a 0, does this mean 'none' or 'unlimited'? -- and tried with both the paths as above, but still no file was created. |
|
| Back to top |
|
|
Hu
Moderator
Joined: 06 Mar 2007
Posts: 21639
|
| Posted: Sun Nov 09, 2008 5:55 pm Post subject: |
|
|
| I get the string unlimited when the core file size is unlimited. I think a 0 means that a core file can be at most zero bytes long. In this case, no core will be written. |
|
| Back to top |
|
|
Havin_it
Veteran
Joined: 17 Jul 2005
Posts: 1247
Location: Edinburgh, UK
|
| Posted: Mon Nov 10, 2008 1:52 am Post subject: |
|
|
OK, definitely have it set to unlimited now, but still no output  |
|
| Back to top |
|
|
|
Networking & Security
