Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[SOLVED] Segfault when trying to unmount davfs
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Networking & Security
View previous topic :: View next topic  
Author Message
gr3m1in
n00b
n00b


Joined: 22 Mar 2010
Posts: 31
Location: Poland

PostPosted: Tue Apr 30, 2019 2:00 pm    Post subject: [SOLVED] Segfault when trying to unmount davfs Reply with quote

Hi all,

I'm having the same problem as posted here half a month before
https://archives.gentoo.org/gentoo-user/message/ba28f9376768d3b0164bbb2a66c0a119
however I haven't noticed the exact moment when it started to happen...

Everything works except unmounting (but it was also working fine earlier).
The share is being mounted by a user inside homedir.
Unmounting does not work and segfaults even by root.

Version in use is net-fs/davfs2-1.5.5
Compiled with GCC 8.2.0-r6 against glibc 2.29-r2

Here is strace output for umount.davfs
https://pastebin.com/7wdNVE34

Can anyone please help with the issue?
Thanks in advance!

Kind regards,
gr3m1in


Last edited by gr3m1in on Thu May 09, 2019 2:13 pm; edited 1 time in total
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 15603

PostPosted: Wed May 01, 2019 12:43 am    Post subject: Reply with quote

Please collect a backtrace from the failed process. For crashes, a backtrace is often more directly informative than an strace. You may need to rebuild the affected components with debug symbols first. If you have the same crash as shown in the mailing list thread, then davfs2 is calling glibc in an invalid way. The backtrace should tell us more.
Back to top
View user's profile Send private message
gr3m1in
n00b
n00b


Joined: 22 Mar 2010
Posts: 31
Location: Poland

PostPosted: Wed May 01, 2019 11:10 pm    Post subject: Reply with quote

Thanks for your reply, Hu

I've googled about compiling with debug symbols on gentoo
https://wiki.gentoo.org/wiki/Debugging

However it is still a bit unclear how exactly and which packages should be rebuilt.
Just davfs2? Or glibc also? or everything somehow related to building process (gcc, binutils, ...)?

And which of proposed ways should I do this?
By adding
Quote:
FEATURES="nostrip"
or
Quote:
FEATURES="${FEATURES} splitdebug compressdebug -nostrip"
or by adding "debugsyms" to package.env or even all of this?

Any advice is deeply appreciated!

Kind regards,
gr3m1in
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 15603

PostPosted: Thu May 02, 2019 2:09 am    Post subject: Reply with quote

I use nostrip because it is the simplest. It has the downside that if you want to discard the symbols later to recover space, it's more trouble. If you care about that, use splitdebug instead.

Rebuild whichever package(s) contribute the faulting executable and any shared libraries shown in the backtrace. I'd start with davfs2 and glibc. You may need to add others. Fortunately, you can reproduce this crash easily, so if you don't get everything on the first try, you can easily rebuild and try again. If this were a once-a-week crash with no good steps to reproduce, I would put more emphasis on having the perfect environment standing by waiting for the crash. As is, do those two. If you get any shared libraries in the backtrace that lack symbols, use equery belongs to find the owning package, rebuild that package with debug symbols, and retry. If the backtrace is detailed enough to let you check multiple libraries at once, check all that you can and rebuild each affected package, to reduce the number of round trips.
Back to top
View user's profile Send private message
gr3m1in
n00b
n00b


Joined: 22 Mar 2010
Posts: 31
Location: Poland

PostPosted: Thu May 02, 2019 10:18 am    Post subject: Reply with quote

So what was done:

1. portage env file for turning on nostrip feature
Code:
# cat /etc/portage/env/debug
#FEATURES="nostrip"
FEATURES="${FEATURES} nostrip"


2. list of packages that should use it
Code:
# cat /etc/portage/package.env
sys-libs/glibc              debug
net-fs/davfs2               debug


3. packages rebuild in order
Code:
echo glibc davfs2 | xargs -n 1 emerge --ask=n


After doing so and mounting davfs, no changes were noticed in umount behavior...
Code:
$ umount.davfs ~/dav/
Segmentation fault


dmesg
Code:
[ 3057.923833] umount.davfs[1282]: segfault at 44962000 ip 00007f878b5cf836 sp 00007ffc1df5b198 error 4 in libc-2.29.so[7f878b551000+15a000]
[ 3057.923845] Code: 0f 1f 40 00 66 0f ef c0 66 0f ef c9 66 0f ef d2 66 0f ef db 48 89 f8 48 89 f9 48 81 e1 ff 0f 00 00 48 81 f9 cf 0f 00 00 77 6a <f3> 0f 6f 20 66 0f 74 e0 66 0f d7 d4 85 d2 74 04 0f bc c2 c3 48 83


/var/log/messages
Code:
May  2 12:04:55 book kernel: umount.davfs[1282]: segfault at 44962000 ip 00007f878b5cf836 sp 00007ffc1df5b198 error 4 in libc-2.29.so[7f878b551000+15a000]
May  2 12:04:55 book kernel: Code: 0f 1f 40 00 66 0f ef c0 66 0f ef c9 66 0f ef d2 66 0f ef db 48 89 f8 48 89 f9 48 81 e1 ff 0f 00 00 48 81 f9 cf 0f 00 00 77 6a <f3> 0f 6f 20 66 0f 74 e0 66 0f d7 d4 85 d2 74 04 0f bc c2 c3 48 83


No additional files were created inside /var/log.
Am I doing something wrong?
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 15603

PostPosted: Fri May 03, 2019 2:05 am    Post subject: Reply with quote

gr3m1in wrote:
3. packages rebuild in order
Code:
echo glibc davfs2 | xargs -n 1 emerge --ask=n
There is no ordering dependency here. You could have run emerge --ask --oneshot --verbose sys-libs/glibc net-fs/davfs2.
gr3m1in wrote:
After doing so and mounting davfs, no changes were noticed in umount behavior...
Code:
$ umount.davfs ~/dav/
Segmentation fault
Yes, this is not intended to make it stable. It is only intended to let you debug why it is unstable.
gr3m1in wrote:
Am I doing something wrong?
Yes. You should be looking at the core file, not log files. However, before you can do that, you need it to actually create a core file. Currently, it is not, which is weird. According to your strace output above, its current working directory was still your home directory, which should be fine. Is this program suid or otherwise resistant to dumping (see man 5 proc section on /proc/sys/fs/suid_dumpable for why it might not be dumped)? Do you run it with a ulimit that prevents generating a core file?

Regardless, you could ignore the lack of core file and instead run it under a debugger, so that you can trap when it crashes. Note that using a debugger on a suid program can be a bit tricky. If this program is suid, special steps may be required.
Back to top
View user's profile Send private message
gr3m1in
n00b
n00b


Joined: 22 Mar 2010
Posts: 31
Location: Poland

PostPosted: Fri May 03, 2019 10:24 am    Post subject: Reply with quote

Quote:
Yes, this is not intended to make it stable. It is only intended to let you debug why it is unstable.

Yes, I do understand that it shouldn't make is stable.
I mean that the output hasn't changed as well as logs.

However after you mentioned ulimit I found the way it should be used https://wiki.gentoo.org/wiki/Project:Quality_Assurance/Backtraces#Core_dumps
and after setting
Code:
ulimit -c unlimited

the core dump has been created
Code:
$ umount.davfs ~/dav/
Segmentation fault (core dumped)

and the core file appeared inside homedir.

As it is described via link above, I declared a helper function
Code:
$ gdb_get_backtrace() {
>     local exe=$1
>     local core=$2
>
>     gdb ${exe} \
>         --core ${core} \
>         --batch \
>         --quiet \
>         -ex "thread apply all bt full" \
>         -ex "quit"
> }

and run it with corresponding parameters
Code:
$ gdb_get_backtrace /usr/sbin/umount.davfs ./core
[New LWP 23492]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `umount.davfs /home/gr3m1in/dav/'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f6241be0836 in __strlen_sse2 () from /lib64/libc.so.6

Thread 1 (Thread 0x7f623f706740 (LWP 23492)):
#0  0x00007f6241be0836 in __strlen_sse2 () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f6241d20740 in ne_concat () from /usr/lib64/libneon.so.27
No symbol table info available.
#2  0x000056481dcd9453 in main ()
No symbol table info available.


Is it ok that "No symbol table info available" is shown for /lib64/libc.so.6 since glibc was rebuilt with nostrip feature?
And should I also rebuild net-libs/neon with nostrip?
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 15603

PostPosted: Sat May 04, 2019 12:44 am    Post subject: Reply with quote

gr3m1in wrote:
Is it ok that "No symbol table info available" is shown for /lib64/libc.so.6 since glibc was rebuilt with nostrip feature?
It's not expected, but the output you have so far is probably good enough for glibc.
gr3m1in wrote:
And should I also rebuild net-libs/neon with nostrip?
Yes, and davfs2 itself too please. If I were to guess from the available data, I would say that ne_concat was called with an invalid pointer, which then faulted in glibc when Neon tried to compute the length of the string. Having reviewed the likely callers of ne_concat in davfs (https://fossies.org/linux/www/davfs2-1.5.5.tar.gz/davfs2-1.5.5/src/umount_davfs.c?m=t), I see several trivially obvious memory leaks, unsafe uses of system, and a buggy way of waiting for the process. Now that I know how this tool works, I would not run it without a code audit and several patches.
Back to top
View user's profile Send private message
gr3m1in
n00b
n00b


Joined: 22 Mar 2010
Posts: 31
Location: Poland

PostPosted: Wed May 08, 2019 9:31 am    Post subject: Reply with quote

I have added -g key to cflags and cxxflags in debug env file so it now looks like this
Code:
# cat /etc/portage/env/debug
CFLAGS="${CFLAGS} -g"
CXXFLAGS="${CXXFLAGS} -g"
FEATURES="${FEATURES} nostrip"

and rebuilt the following packages again
Code:
# cat /etc/portage/package.env
sys-libs/glibc              debug
net-libs/neon               debug
net-fs/davfs2               debug

so now after trying to unmount the share the backtrace looks more informative
Code:
$ gdb_get_backtrace /usr/sbin/umount.davfs ./core
[New LWP 13150]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `umount.davfs /home/gr3m1in/dav/'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:120
120     ../sysdeps/x86_64/multiarch/../strlen.S: No such file or directory.

Thread 1 (Thread 0x7f757f276740 (LWP 13150)):
#0  __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:120
No locals.
#1  0x00007f7581890740 in count_concat (ap=0x7ffd33e19fd0) at ne_string.c:132
        total = 0
        next = 0x45bec000 <error: Cannot access memory at address 0x45bec000>
        total = <optimized out>
        next = <optimized out>
#2  ne_concat (str=0x55cfd85f0095 "ps -p ") at ne_string.c:179
        ap = {{gp_offset = 16, fp_offset = 32629, overflow_arg_area = 0x7ffd33e1a050, reg_save_area = 0x7ffd33e19ff0}}
        total = <optimized out>
        slen = 6
        ret = <optimized out>
#3  0x000055cfd85ef453 in main (argc=<optimized out>, argv=0x7ffd33e1a198) at umount_davfs.c:163
        short_options = 0x55cfd85f001d "Vhflnrt:v"
        options = {{name = 0x55cfd85f00a0 "version", has_arg = 0, flag = 0x0, val = 86}, {name = 0x55cfd85f00a8 "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}}
        o = <optimized out>
        mpoint = <optimized out>
        umount_command = 0x55cfd8d1c020 "umount -i '/home/gr3m1in/dav'"
        m = <optimized out>
        mp = <optimized out>
        pidfile = 0x55cfd8d1c070 "/var/run/mount.davfs/home-gr3m1in-dav.pid"
        pid = 0x45bec000 <error: Cannot access memory at address 0x45bec000>
        file = 0x55cfd8d1c0b0
        ps_command = <optimized out>
        ps_in = <optimized out>
        found = <optimized out>
        n = 94351176694197
        ps_line = 0x7f758187f408 <__exit_funcs_lock> ""

It really looks like an invalid call...
Any advice on how to fix this?
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 15603

PostPosted: Thu May 09, 2019 1:50 am    Post subject: Reply with quote

gr3m1in wrote:
I have added -g key to cflags and cxxflags in debug env file so it now looks like this
This looks correct. The backtrace below agrees.
gr3m1in wrote:
Code:
        umount_command = 0x55cfd8d1c020 "umount -i '/home/gr3m1in/dav'"
Although expected from source inspection, this looks wrong. What if your directory name had a single quote in it?
gr3m1in wrote:
Code:
        pid = 0x45bec000 <error: Cannot access memory at address 0x45bec000>
It really looks like an invalid call...
Any advice on how to fix this?
Agreed, that is invalid. The Makefile does not set a C standard version, and the implementation appears to rely on being built as pre-C99 due to its use of the %a conversion in fscanf for an allocation, not a floating point. Recent versions of gcc default to a newer standard. You could try using the standard-approved %m, or force this package to build in C89 mode. However, as above, my advice from having read the source is to make rather more invasive edits to clean up other bugs and design flaws.
Back to top
View user's profile Send private message
gr3m1in
n00b
n00b


Joined: 22 Mar 2010
Posts: 31
Location: Poland

PostPosted: Thu May 09, 2019 2:12 pm    Post subject: Reply with quote

Many thanks, Hu,
After forcing C89 mode it does work!

Corresponding env file was created with the following contents:
Code:
# cat /etc/portage/env/C89
CFLAGS="${CFLAGS} -std=c89"
CXXFLAGS="${CXXFLAGS} -std=c89"

and then used to build davfs2 package
Code:
cat /etc/portage/package.env
sys-libs/glibc              debug
net-libs/neon               debug
net-fs/davfs2               debug C89

While rebuilding I've checked that the variables were set correctly (just in case)
Code:
x86_64-pc-linux-gnu-gcc -DPROGRAM_NAME=\"mount.davfs\" -DDAV_SYS_CONF_DIR=\"/etc/davfs2\" -DDAV_LOCALSTATE_DIR=\"/var/run\" -DDAV_SYS_RUN=\"/var/run/mount.davfs\" -DDAV_SYS_CACHE=\"/var/cache/davfs2\" -DDAV_SECRETS=\"secrets\" -DDAV_CONFIG=\"davfs2.conf\" -DDAV_CERTS_DIR=\"certs\" -DDAV_CLICERTS_DIR=\"private\" -DDAV_DATA_DIR=\"/usr/share/davfs2\" -DLOCALEDIR=\"/usr/share/locale\" -DDAV_USER=\"nobody\" -DDAV_GROUP=\"davfs2\" -D_FORTIFY_SOURCE=2 -DHAVE_CONFIG_H -I. -I..    -Wall -Werror=format-security -fstack-protector-strong --param=ssp-buffer-size=4 -O2 -pipe -g -std=c89 -I/usr/include/neon -c -o dav_fuse.o dav_fuse.c

and then checked mounting and unmounting
Code:
$ mount dav/
$ umount dav/
/sbin/umount.davfs: waiting while mount.davfs (pid 18095) synchronizes the cache .. OK
$

So now I can confirm that forcing C89 mode for davfs2 solves this problem at the moment.
However, it is not a panacea, and without code remastering a sort of similar problem could appear soon.

Anyway, this particular problem is now solved.
Thank you again, Hu!

Kind regards,
gr3m1in
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Networking & Security All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum