Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
"Accessing a corrupted shared library" [CAN'T REPRODUCE]
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
sliwowitz
Apprentice
Apprentice


Joined: 21 Jan 2005
Posts: 194
Location: Europe/Prague

PostPosted: Thu Jan 18, 2018 3:22 pm    Post subject: "Accessing a corrupted shared library" [CAN'T REPR Reply with quote

I tried to update our production machine today to migrate it to the new 17.0 profiles. I was following the guide https://www.gentoo.org/support/news-items/2017-11-30-new-17-profiles.html

During the glibc compilation, I suddenly got a bunch of errors like:
Code:
OSError: [Errno 80] Accessing a corrupted shared library: b'/bin/bash'
[Errno 80] Accessing a corrupted shared library: b'/usr/bin/sandbox':
   /usr/bin/sandbox /usr/lib/portage/python3.4/misc-functions.sh die_hooks
Traceback (most recent call last):
  File "/usr/lib64/python3.4/site-packages/portage/process.py", line 321, in spawn
    unshare_net, unshare_ipc, cgroup)
  File "/usr/lib64/python3.4/site-packages/portage/process.py", line 519, in _exec
    os.execve(binary, myargs, env)
  File "/usr/lib64/python3.4/site-packages/portage/__init__.py", line 250, in __call__
    rval = self._func(*wrapped_args, **wrapped_kwargs)
OSError: [Errno 80] Accessing a corrupted shared library: b'/usr/bin/sandbox'
 * The ebuild phase 'die_hooks' has exited unexpectedly. This type of


I'm still logged in in one of the terminals, but I can't run anything at all:
Code:
epsrv ~ # ls
-bash: /bin/ls: Accessing a corrupted shared library


Any tips on how to recover (apart from restoring the machine from a backup)
_________________
The question of whether Machines Can Think... is about as relevant as the question of whether Submarines Can Swim.
-- Edsger Dijkstra


Last edited by sliwowitz on Mon Jan 22, 2018 11:32 am; edited 3 times in total
Back to top
View user's profile Send private message
mike155
Guru
Guru


Joined: 17 Sep 2010
Posts: 537
Location: Frankfurt, Germany

PostPosted: Thu Jan 18, 2018 3:37 pm    Post subject: Reply with quote

This looks pretty bad...

First of all: do NOT log out. Does 'busybox ls' work? If it works, try 'busybox sh' to get a (somewhat limited) shell. Basic commands like ls, rm, mkdir should work again, because busybox uses its own compiled-in versions of these commands.
Back to top
View user's profile Send private message
sliwowitz
Apprentice
Apprentice


Joined: 21 Jan 2005
Posts: 194
Location: Europe/Prague

PostPosted: Thu Jan 18, 2018 3:44 pm    Post subject: Reply with quote

OK. Busybox works
_________________
The question of whether Machines Can Think... is about as relevant as the question of whether Submarines Can Swim.
-- Edsger Dijkstra
Back to top
View user's profile Send private message
mike155
Guru
Guru


Joined: 17 Sep 2010
Posts: 537
Location: Frankfurt, Germany

PostPosted: Thu Jan 18, 2018 3:55 pm    Post subject: Reply with quote

The next step: please try to find out what went wrong. Migration to 17.0 is complex, but it worked flawlessly for most people. So something exceptional must have happened in your case. Do you have any idea, what this could have been? Check log files, check the migration guide again.
Back to top
View user's profile Send private message
sliwowitz
Apprentice
Apprentice


Joined: 21 Jan 2005
Posts: 194
Location: Europe/Prague

PostPosted: Thu Jan 18, 2018 4:24 pm    Post subject: Reply with quote

I have migrated about 10 servers without problems already. I have no idea what's different with this one. The emerge logs show nothing unusal:
Code:

1516284864:  *** emerge --oneshot --jobs=4 --load-average=4.0 sandbox
1516284865:  >>> emerge (1 of 1) sys-apps/sandbox-2.12 to /
1516284865:  === (1 of 1) Cleaning (sys-apps/sandbox-2.12::/usr/portage/sys-apps/sandbox/sandbox-2.12.ebuild)
1516284865:  === (1 of 1) Compiling/Merging (sys-apps/sandbox-2.12::/usr/portage/sys-apps/sandbox/sandbox-2.12.ebuild)
1516284908:  === (1 of 1) Merging (sys-apps/sandbox-2.12::/usr/portage/sys-apps/sandbox/sandbox-2.12.ebuild)
1516284909:  >>> AUTOCLEAN: sys-apps/sandbox:0
1516284909:  === Unmerging... (sys-apps/sandbox-2.12)
1516284910:  >>> unmerge success: sys-apps/sandbox-2.12
1516284911:  === (1 of 1) Post-Build Cleaning (sys-apps/sandbox-2.12::/usr/portage/sys-apps/sandbox/sandbox-2.12.ebuild)
1516284911:  ::: completed emerge (1 of 1) sys-apps/sandbox-2.12 to /
1516284911:  *** Finished. Cleaning up...
1516284911:  *** exiting successfully.
1516284911:  *** terminating.
1516284912: Started emerge on: Jan 18, 2018 15:15:11
1516284912:  *** emerge --load-average=4.0 --jobs=4 --oneshot libtool
1516284913:  >>> emerge (1 of 1) sys-devel/libtool-2.4.6-r3 to /
1516284913:  === (1 of 1) Cleaning (sys-devel/libtool-2.4.6-r3::/usr/portage/sys-devel/libtool/libtool-2.4.6-r3.ebuild)
1516284913:  === (1 of 1) Compiling/Merging (sys-devel/libtool-2.4.6-r3::/usr/portage/sys-devel/libtool/libtool-2.4.6-r3.ebuild)
1516284939:  === (1 of 1) Merging (sys-devel/libtool-2.4.6-r3::/usr/portage/sys-devel/libtool/libtool-2.4.6-r3.ebuild)
1516284940:  >>> AUTOCLEAN: sys-devel/libtool:2
1516284940:  === Unmerging... (sys-devel/libtool-2.4.6-r3)
1516284941:  >>> unmerge success: sys-devel/libtool-2.4.6-r3
1516284942:  === (1 of 1) Post-Build Cleaning (sys-devel/libtool-2.4.6-r3::/usr/portage/sys-devel/libtool/libtool-2.4.6-r3.ebuild)
1516284942:  ::: completed emerge (1 of 1) sys-devel/libtool-2.4.6-r3 to /
1516284942:  *** Finished. Cleaning up...
1516284942:  *** exiting successfully.
1516284945:  *** terminating.
1516285021: Started emerge on: Jan 18, 2018 15:17:00
1516285021:  *** emerge --jobs=4 --oneshot --load-average=4.0 sys-devel/gcc:6.4.0
1516285026:  >>> emerge (1 of 1) sys-devel/gcc-6.4.0-r1 to /
1516285026:  === (1 of 1) Cleaning (sys-devel/gcc-6.4.0-r1::/usr/portage/sys-devel/gcc/gcc-6.4.0-r1.ebuild)
1516285026:  === (1 of 1) Compiling/Merging (sys-devel/gcc-6.4.0-r1::/usr/portage/sys-devel/gcc/gcc-6.4.0-r1.ebuild)
1516285278:  *** Finished. Cleaning up...
1516285278:  *** exiting unsuccessfully with status '1'.
1516285278:  *** terminating.
1516285327: Started emerge on: Jan 18, 2018 15:22:06
1516285327:  *** emerge --jobs=4 --oneshot --load-average=4.0 gcc
1516285332:  >>> emerge (1 of 1) sys-devel/gcc-6.4.0-r1 to /
1516285332:  === (1 of 1) Cleaning (sys-devel/gcc-6.4.0-r1::/usr/portage/sys-devel/gcc/gcc-6.4.0-r1.ebuild)
1516285332:  === (1 of 1) Compiling/Merging (sys-devel/gcc-6.4.0-r1::/usr/portage/sys-devel/gcc/gcc-6.4.0-r1.ebuild)
1516287348:  === (1 of 1) Merging (sys-devel/gcc-6.4.0-r1::/usr/portage/sys-devel/gcc/gcc-6.4.0-r1.ebuild)
1516287351:  >>> AUTOCLEAN: sys-devel/gcc:6.4.0
1516287354:  === (1 of 1) Post-Build Cleaning (sys-devel/gcc-6.4.0-r1::/usr/portage/sys-devel/gcc/gcc-6.4.0-r1.ebuild)
1516287354:  ::: completed emerge (1 of 1) sys-devel/gcc-6.4.0-r1 to /
1516287354:  *** Finished. Cleaning up...
1516287354:  *** exiting successfully.
1516287354:  *** terminating.
1516287586: Started emerge on: Jan 18, 2018 15:59:45
1516287586:  *** emerge --load-average=4.0 --jobs=4 --oneshot sys-devel/binutils
1516287588:  >>> emerge (1 of 1) sys-devel/binutils-2.29.1-r1 to /
1516287588:  === (1 of 1) Cleaning (sys-devel/binutils-2.29.1-r1::/usr/portage/sys-devel/binutils/binutils-2.29.1-r1.ebuild)
1516287590:  === (1 of 1) Compiling/Merging (sys-devel/binutils-2.29.1-r1::/usr/portage/sys-devel/binutils/binutils-2.29.1-r1.ebuild)
1516287709:  === (1 of 1) Merging (sys-devel/binutils-2.29.1-r1::/usr/portage/sys-devel/binutils/binutils-2.29.1-r1.ebuild)
1516287710:  >>> AUTOCLEAN: sys-devel/binutils:2.29.1
1516287712:  === (1 of 1) Post-Build Cleaning (sys-devel/binutils-2.29.1-r1::/usr/portage/sys-devel/binutils/binutils-2.29.1-r1.ebuild)
1516287712:  ::: completed emerge (1 of 1) sys-devel/binutils-2.29.1-r1 to /
1516287712:  *** Finished. Cleaning up...
1516287712:  *** exiting successfully.
1516287712:  *** terminating.
1516287757: Started emerge on: Jan 18, 2018 16:02:37
1516287757:  *** emerge --jobs=4 --load-average=4.0 --oneshot sys-libs/glibc
1516287761:  >>> emerge (1 of 1) sys-libs/glibc-2.25-r9 to /
1516287761:  === (1 of 1) Cleaning (sys-libs/glibc-2.25-r9::/usr/portage/sys-libs/glibc/glibc-2.25-r9.ebuild)
1516287773:  === (1 of 1) Compiling/Merging (sys-libs/glibc-2.25-r9::/usr/portage/sys-libs/glibc/glibc-2.25-r9.ebuild)
1516288191:  === (1 of 1) Merging (sys-libs/glibc-2.25-r9::/usr/portage/sys-libs/glibc/glibc-2.25-r9.ebuild)
1516288194:  >>> AUTOCLEAN: sys-libs/glibc:2.2
1516288194:  === Unmerging... (sys-libs/glibc-2.23-r3)
1516288201:  >>> unmerge success: sys-libs/glibc-2.23-r3
1516288202:  === (1 of 1) Post-Build Cleaning (sys-libs/glibc-2.25-r9::/usr/portage/sys-libs/glibc/glibc-2.25-r9.ebuild)
1516288202:  ::: completed emerge (1 of 1) sys-libs/glibc-2.25-r9 to /
1516288202:  *** Finished. Cleaning up...
1516288205:  *** exiting unsuccessfully with status '1'.
1516288206:  *** terminating.


I don't remember the detail about the first unsuccessful gcc emerge. This is what I see in root's .bash_history
Code:

. /etc/profile
emerge -1 sys-devel/libtool
emerge -1 libtool sandbox
emerge -1 sandbox && emerge -1 libtool
eselect profile list
eselect profile set default/linux/amd64/17.0/hardened
emerge -1 sys-devel/gcc:6.4.0 && emerge -1 sys-devel/binutils && emerge -1 sys-libs/glibc
emerge -1 sys-devel/gcc:6.4.0 && emerge -1 sys-devel/binutils && emerge -1 sys-libs/glibc


The emerge log desn't seem to contain anything useful apart from a bunch of repeating meassages about bash and sandbox accessing a "corrupted library" right after the install phase:
Code:


... snipped ...

>>> Completed installing glibc-2.25-r9 into /var/tmp/portage/sys-libs/glibc-2.25-r9/image/

 * Final size of build directory: 590012 KiB
 * Final size of installed tree: 75756 KiB

making executable: usr/lib/libc.so
making executable: usr/lib/libpthread.so
making executable: usr/lib64/libc.so
making executable: usr/lib64/libm.so
making executable: usr/lib64/libpthread.so
ecompressdir: bzip2 -9 /usr/share/man
ecompressdir: bzip2 -9 /usr/share/info
ecompressdir: bzip2 -9 /usr/share/doc
 * Defaulting /etc/host.conf:multi to on
 *
 * Installation of a symlink is blocked by a directory:
 *   '/usr/lib/locale'
 * This symlink will be merged with a different name:
 *   '/usr/lib/locale.backup.0000'
 *
 * The ebuild phase 'postrm' has exited unexpectedly. This type of behavior
 * is known to be triggered by things such as failed variable assignments
 * (bug #190128) or bad substitution errors (bug #200313). Normally, before
 * exiting, bash should have displayed an error message above. If bash did
 * not produce an error message above, it's possible that the ebuild has
 * called `exit` when it should have called `die` instead. This behavior
 * may also be triggered by a corrupt bash binary or a hardware problem
 * such as memory or cpu malfunction. If the problem is not reproducible or
 * it appears to occur randomly, then it is likely to be triggered by a
 * hardware problem. If you suspect a hardware problem then you should try
 * some basic hardware diagnostics such as memtest. Please do not report
 * this as a bug unless it is consistently reproducible and you are sure
 * that your bash binary and hardware are functioning properly.
 * The ebuild phase 'die_hooks' has exited unexpectedly. This type of
 * behavior is known to be triggered by things such as failed variable
 * assignments (bug #190128) or bad substitution errors (bug #200313).
 * Normally, before exiting, bash should have displayed an error message
 * above. If bash did not produce an error message above, it's possible
 * that the ebuild has called `exit` when it should have called `die`
 * instead. This behavior may also be triggered by a corrupt bash binary or
 * a hardware problem such as memory or cpu malfunction. If the problem is
 * not reproducible or it appears to occur randomly, then it is likely to
 * be triggered by a hardware problem. If you suspect a hardware problem
 * then you should try some basic hardware diagnostics such as memtest.
 * Please do not report this as a bug unless it is consistently
 * reproducible and you are sure that your bash binary and hardware are
 * functioning properly.
[Errno 80] Accessing a corrupted shared library: b'/bin/bash':
   /bin/bash -c /usr/lib/portage/python3.4/ebuild.sh postinst
Traceback (most recent call last):
  File "/usr/lib64/python3.4/site-packages/portage/process.py", line 321, in spawn
    unshare_net, unshare_ipc, cgroup)
  File "/usr/lib64/python3.4/site-packages/portage/process.py", line 519, in _exec
    os.execve(binary, myargs, env)
  File "/usr/lib64/python3.4/site-packages/portage/__init__.py", line 250, in __call__
    rval = self._func(*wrapped_args, **wrapped_kwargs)
OSError: [Errno 80] Accessing a corrupted shared library: b'/bin/bash'
 * The ebuild phase 'postinst' has exited unexpectedly. This type of
 * behavior is known to be triggered by things such as failed variable
 * assignments (bug #190128) or bad substitution errors (bug #200313).
 * Normally, before exiting, bash should have displayed an error message
 * above. If bash did not produce an error message above, it's possible
 * that the ebuild has called `exit` when it should have called `die`
 * instead. This behavior may also be triggered by a corrupt bash binary or
 * a hardware problem such as memory or cpu malfunction. If the problem is
 * not reproducible or it appears to occur randomly, then it is likely to
 * be triggered by a hardware problem. If you suspect a hardware problem
 * then you should try some basic hardware diagnostics such as memtest.
 * Please do not report this as a bug unless it is consistently
 * reproducible and you are sure that your bash binary and hardware are
 * functioning properly.
[Errno 80] Accessing a corrupted shared library: b'/usr/bin/sandbox':
   /usr/bin/sandbox /usr/lib/portage/python3.4/misc-functions.sh die_hooks
Traceback (most recent call last):
  File "/usr/lib64/python3.4/site-packages/portage/process.py", line 321, in spawn
    unshare_net, unshare_ipc, cgroup)
  File "/usr/lib64/python3.4/site-packages/portage/process.py", line 519, in _exec
    os.execve(binary, myargs, env)
  File "/usr/lib64/python3.4/site-packages/portage/__init__.py", line 250, in __call__
    rval = self._func(*wrapped_args, **wrapped_kwargs)
OSError: [Errno 80] Accessing a corrupted shared library: b'/usr/bin/sandbox'

... snipped a bunch of similar messages ...

_________________
The question of whether Machines Can Think... is about as relevant as the question of whether Submarines Can Swim.
-- Edsger Dijkstra


Last edited by sliwowitz on Fri Jan 19, 2018 8:19 am; edited 2 times in total
Back to top
View user's profile Send private message
mike155
Guru
Guru


Joined: 17 Sep 2010
Posts: 537
Location: Frankfurt, Germany

PostPosted: Thu Jan 18, 2018 4:34 pm    Post subject: Reply with quote

I'm a bit surprised by 'python3.4'. When did you do the last 'emerge --update --deep @world' and 'emerge --depclean' on that machine?
Back to top
View user's profile Send private message
Jaglover
Watchman
Watchman


Joined: 29 May 2005
Posts: 6397
Location: Saint Amant, Acadiana

PostPosted: Thu Jan 18, 2018 4:38 pm    Post subject: Reply with quote

Anything corrupted I'd start with filesystem and hard drive diagnostics.
_________________
Please learn how to denote units correctly!
Back to top
View user's profile Send private message
sliwowitz
Apprentice
Apprentice


Joined: 21 Jan 2005
Posts: 194
Location: Europe/Prague

PostPosted: Thu Jan 18, 2018 5:01 pm    Post subject: Reply with quote

mike155 wrote:
I'm a bit surprised by 'python3.4'. When did you do the last 'emerge --update --deep @world' and 'emerge --depclean' on that machine?

I'm keeping this version because of a python web application written for 3.4. It might work with newer versions, but we didn't fully test it yet. The server runs the app through apache+mod_wsgi which has to be compiled for the correct python version.
Jaglover wrote:
Anything corrupted I'd start with filesystem and hard drive diagnostics.

It's a virtual machine in a large VMWare cluster. If it is a hard drive / filesystem issue, we would have to take it down for maintenance. In that case I would restore it from the backup anyways. I was more interested in whether it would be possible to get it back to working state without the need to stop/restart it, as those applications (like apache and postgres) which were already running before I started the update are still running now.
_________________
The question of whether Machines Can Think... is about as relevant as the question of whether Submarines Can Swim.
-- Edsger Dijkstra
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 6804

PostPosted: Thu Jan 18, 2018 9:00 pm    Post subject: Reply with quote

did you try look at /bin/bash if the file is still a valid ELF64?
some user with zfs had corruption (file zeroed) with well, cannot remember but some bad zfs tools. (if you use zfs dig the forum for such issue, or gentoo bugzilla for the fix).
Back to top
View user's profile Send private message
sliwowitz
Apprentice
Apprentice


Joined: 21 Jan 2005
Posts: 194
Location: Europe/Prague

PostPosted: Fri Jan 19, 2018 2:08 pm    Post subject: Reply with quote

The partitions are ext4. I have lost the ssh connection overnight, and the login wouldn't work (beacuse of that "Accessing a corrupted shared library" error). In the end, I restored the whole machine from a backup. Now I'm gonna try the update again. BTW, last update of this machine was some 221 days ago, so no wonder it's a bit of an adventure :-)
_________________
The question of whether Machines Can Think... is about as relevant as the question of whether Submarines Can Swim.
-- Edsger Dijkstra
Back to top
View user's profile Send private message
mike155
Guru
Guru


Joined: 17 Sep 2010
Posts: 537
Location: Frankfurt, Germany

PostPosted: Fri Jan 19, 2018 2:19 pm    Post subject: Reply with quote

I'm glad that you could restore your machine from a backup. :-)

Maybe you should first update your machine using 'emerge --update --deep --newuse @world; emerge --depclean' before you switch to the new profile.

Good luck!
Back to top
View user's profile Send private message
sliwowitz
Apprentice
Apprentice


Joined: 21 Jan 2005
Posts: 194
Location: Europe/Prague

PostPosted: Mon Jan 22, 2018 11:45 am    Post subject: Reply with quote

mike155 wrote:
I'm glad that you could restore your machine from a backup. :-)

Yes - huge thanks to our IT department, who back up the whole VM daily.

mike155 wrote:
Maybe you should first update your machine using 'emerge --update --deep --newuse @world; emerge --depclean' before you switch to the new profile.

That's exactly how I did it this time. With the other machines, I first went to change the profile then the updates because I didn't want to recompile too much stuff twice, but the other machines were in a better state with much more regular updates than this one. Anyways, now I did a full system update first, then the profile change according to the guide, and everything went smoothly. I don't know if the world update helped, or if it was some HW / filesystem failure before, I guess I won't find out.

On a side note - seeing how unexpectedly can the things go bad, I actually bought an external HDD for my home computer, and set up regular backups after some 12 years of telling everybody how important it is to do these, and not actaully ever getting around to doing it at all :D
_________________
The question of whether Machines Can Think... is about as relevant as the question of whether Submarines Can Swim.
-- Edsger Dijkstra
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum