| View previous topic :: View next topic |
| Author |
Message |
midway
Apprentice
Joined: 01 Jun 2012
Posts: 181
Location: somewhr in the blighty
|
 Posted: Thu Aug 16, 2012 10:56 am Post subject: [solved] IOError: Errno 28 No space left on device |
 |
|
Hello All,
On an emerge system I got 55 updates which failed complaining about no disk space left. I am attaching the failed error below. Someone mentioned on net to increase the inode options. Could you describe the steps or point me to a guide at-least? I am in the middle of updating this system.
| Code: |
* To avoid automounting and auto(un)installing with /boot,
* just export the DONT_MOUNT_BOOT variable.
*
*
* memtest has been installed in /boot/memtest86plus/
* You may wish to update your bootloader configs
* by adding these lines:
* - For grub2 just run grub-mkconfig, a configuration file is installed
* as /etc/grub/39_memtest86+
* - For grub legacy: (replace '?' with correct numbers for your boot partition)
* > title=memtest86+
* > root (hd?,?)
* > kernel /boot/memtest86plus/memtest
* - For lilo:
* > image = /boot/memtest86plus/memtest
* > label = memtest86+
*
* Messages for package dev-libs/gmp-5.0.5:
* Old versions of installed libraries were detected on your system.
* In order to avoid breaking packages that depend on these old libs,
* the libraries are not being removed. You need to run revdep-rebuild
* in order to remove these old dependencies. If you do not have this
* helper program, simply emerge the 'gentoolkit' package.
* # revdep-rebuild --library '/usr/lib/libgmp.so.3' && rm '/usr/lib/libgmp.so.3'
* Messages for package sys-fs/squashfs-tools-4.2-r1:
* This version of mksquashfs requires a 2.6.29 kernel or better
* XZ support requires a 2.6.38 kernel or better
* Messages for package dev-libs/mpfr-3.1.1:
* Old versions of installed libraries were detected on your system.
* In order to avoid breaking packages that depend on these old libs,
* the libraries are not being removed. You need to run revdep-rebuild
* in order to remove these old dependencies. If you do not have this
* helper program, simply emerge the 'gentoolkit' package.
* # revdep-rebuild --library '/usr/lib/libmpfr.so.1' && rm '/usr/lib/libmpfr.so.1'
* Messages for package dev-php/adodb-5.17:
* ADODB requires some form of SQL or ODBC support in your PHP.
* Messages for package dev-lang/perl-5.16.1:
* UPDATE THE PERL MODULES:
* After updating dev-lang/perl you must reinstall
* the installed perl modules.
* Use: perl-cleaner --all
* You must reinstall perl-core/ExtUtils-ParseXS !
* The complete build log is located at '/var/tmp/portage/dev-lang/perl-5.16.1/temp/build.log'.
* The ebuild environment file is located at '/var/tmp/portage/dev-lang/perl-5.16.1/temp/environment'.
* Working directory: '/var/tmp/portage/dev-lang/perl-5.16.1/work/perl-5.16.1'
* S: '/var/tmp/portage/dev-lang/perl-5.16.1/work/perl-5.16.1'
* Regenerating GNU info directory index...
* Processed 109 info files; 46 errors.
install-info: /usr/share/info/dir: empty file
install-info: /usr/share/info/dir: empty file
[this repeated 46 times]
Traceback (most recent call last):
File "/usr/bin/emerge", line 50, in <module>
retval = emerge_main()
File "/usr/lib/portage/pym/_emerge/main.py", line 2073, in emerge_main
trees, mtimedb, retval)
File "/usr/lib/portage/pym/_emerge/main.py", line 402, in post_emerge
mtimedb.commit()
File "/usr/lib/portage/pym/portage/util/mtimedb.py", line 122, in commit
encoding=_encodings['repo.content'], errors='strict'))
File "/usr/lib/portage/pym/portage/util/__init__.py", line 1185, in write
return f.write(s)
IOError: [Errno 28] No space left on device
# revdep-rebuild
/usr/bin/revdep-rebuild: line 128: cannot create temp file for here-document: No space left on device
* Configuring search environment for revdep-rebuild
/usr/bin/revdep-rebuild: line 1080: cannot create temp file for here-document: No space left on device
/usr/bin/revdep-rebuild: line 1089: cannot create temp file for here-document: No space left on device
/usr/bin/revdep-rebuild: line 1090: cannot create temp file for here-document: No space left on device
/usr/bin/revdep-rebuild: line 1091: cannot create temp file for here-document: No space left on device
/usr/bin/revdep-rebuild: line 1100: cannot create temp file for here-document: No space left on device
* No search defined -- this is a bug.
|
Here are more information:
| Code: |
# cat /etc/fstab
/dev/sda1 /boot xfs defaults,noatime 0 1
/dev/sda2 swap swap sw 0 0
/dev/sda3 / xfs noatime 0 1
/dev/cdrom2 /mnt/cdrom auto defaults,noauto 0 0
shm /dev/shm tmpfs nodev,nosuid,noexec 0 0
# du -h --one-file-system --max-depth=1
0 ./.ICE-unix
0 ./.X11-unix
0 ./.winbindd
0 ./hsperfdata_root
0 .
# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
rootfs 536720 499207 37513 94% /
/dev/sda3 536720 499207 37513 94% /
tmpfs 129132 168 128964 1% /run
rc-svcdir 129132 76 129056 1% /lib/rc/init.d
cgroup_root 129132 4 129128 1% /sys/fs/cgroup
udev 129132 500 128632 1% /dev
shm 129132 1 129131 1% /dev/shm
/dev/sda1 152576 47 152529 1% /boot
# df -hT
Filesystem Type Size Used Avail Use% Mounted on
rootfs rootfs 48G 48G 20K 100% /
/dev/sda3 xfs 48G 48G 20K 100% /
tmpfs tmpfs 505M 140K 505M 1% /run
rc-svcdir tmpfs 1.0M 100K 924K 10% /lib/rc/init.d
cgroup_root tmpfs 10M 0 10M 0% /sys/fs/cgroup
udev tmpfs 10M 4.0K 10M 1% /dev
shm tmpfs 505M 0 505M 0% /dev/shm
/dev/sda1 xfs 145M 43M 103M 30% /boot
# du -hx --max-depth=1 /
4.0K /boot
8.8M /bin
0 /dev
15M /etc
195M /lib
0 /mnt
205M /opt
0 /tmp
0 /sys
40G /var
6.5G /usr
0 /proc
64M /home
9.1M /sbin
788M /root
12K /.gnupg
0 /run
48G /
|
Last edited by midway on Fri Aug 17, 2012 11:04 am; edited 1 time in total |
|
| Back to top |
|
 |
aCOSwt
Bodhisattva
Joined: 19 Oct 2007
Posts: 2537
Location: Hilbert space
|
| Posted: Thu Aug 16, 2012 11:39 am Post subject: Re: IOError: [Errno 28] No space left on device |
|
|
| nitish.anand wrote: | Someone mentioned on net to increase the inode options.
| Code: |
# df -hT
Filesystem Type Size Used Avail Use% Mounted on
rootfs rootfs 48G 48G 20K 100% /
|
|
I am afraid that the number of inodes available is not the most significant problem.
Your root filesystem is actually full.
=> Considering its size of 48G, I believe that it should be just easy to find a couple of things to just rm.
| nitish.anand wrote: | | Code: | # du -hx --max-depth=1 /
40G /var |
|
Start your invertigations from there. 40G in /var is far far far too much.
Unless you get tons of mails / printlogs etc...
_________________
|
|
| Back to top |
|
|
midway
Apprentice
Joined: 01 Jun 2012
Posts: 181
Location: somewhr in the blighty
|
| Posted: Thu Aug 16, 2012 12:06 pm Post subject: |
|
|
thx aCOSwt. here seems to be the culprits:
| Code: |
/var/log:
-rw-rw-r-- 1 root utmp 2723712 Aug 15 12:23 wtmp
-rw-r--r-- 1 root root 292876 Aug 15 12:23 lastlog
-rw-rw---- 1 portage portage 1326645 Aug 15 17:00 emerge.log
-rw-rw---- 1 portage portage 1226645 Aug 15 17:00 emerge-fetch.log
-rw-r--r-- 1 root root 151054 Feb 10 2012 retropds.log
-rw------- 1 root root 672128 Aug 15 12:23 tallylog
|
I have rm the lastlog, wtmp and emerge-fetch.log. AFAIK emerge.log is a good thing to keep? Also the new output looks the same:
| Code: |
log # du -hx --max-depth=1 /
4.0K /boot
8.8M /bin
0 /dev
15M /etc
195M /lib
0 /mnt
205M /opt
0 /tmp
0 /sys
40G /var
6.5G /usr
0 /proc
64M /home
9.1M /sbin
788M /root
12K /.gnupg
0 /run
48G /
|
I am puzzled now. Also can the contents of /var/tmp/portage go for good?
| Code: |
tmp # ls -al
total 8
drwxrwxrwt 7 root root 96 Aug 15 15:10 .
drwxr-xr-x 15 root root 4096 Aug 15 14:57 ..
drwx--x--x 3 root root 16 Nov 8 2009 Month_9-2
drwxrwxr-x 2 portage portage 6 Mar 27 12:59 binpkgs
drwxr-xr-x 2 root root 146 Feb 7 2011 genkernel
-rw------- 1 root root 1977 Aug 8 16:53 nagios-044_0
drwxrwxrwt 2 root root 38 Jan 4 2011 pear
drwxrwxr-x 6 portage portage 71 Aug 15 17:00 portage
tmp # cd portage
portage # ls -al
total 0
drwxrwxr-x 6 portage portage 71 Aug 15 17:00 .
drwxrwxrwt 7 root root 96 Aug 15 15:10 ..
drwxrwxr-x 2 portage portage 6 Aug 15 16:53 ._unmerge_
drwxrwxr-x 3 portage portage 24 Aug 15 17:00 dev-lang
drwxrwxr-x 3 portage portage 33 Aug 14 17:16 dev-vcs
drwxrwxr-x 3 portage portage 29 Aug 15 16:42 net-analyzer
|
|
|
| Back to top |
|
|
aCOSwt
Bodhisattva
Joined: 19 Oct 2007
Posts: 2537
Location: Hilbert space
|
| Posted: Thu Aug 16, 2012 12:26 pm Post subject: |
|
|
| nitish.anand wrote: |
I have rm the lastlog, wtmp and emerge-fetch.log. AFAIK emerge.log is a good thing to keep? Also the new output looks the same |
Of course the output looks the same. You are having 40 gigs in /var and just removed a couple of megs.
Culprits are elsewhere under /var.
| nitish.anand wrote: |
I am puzzled now. Also can the contents of /var/tmp/portage go for good?
|
If you are not currently emerging anything and do not want to keep the tracks of failed builds then yes, you can safely remove the content of /var/tmp/portage.
_________________
|
|
| Back to top |
|
|
midway
Apprentice
Joined: 01 Jun 2012
Posts: 181
Location: somewhr in the blighty
|
| Posted: Thu Aug 16, 2012 1:17 pm Post subject: |
|
|
Okay, I think I have finally drilled down:
| Code: |
# du -a /var | sort -n -r | head -n 10
41627668 /var
40165284 /var/nagios
39616724 /var/nagios/archives
980656 /var/lib
963064 /var/lib/mysql
530932 /var/nagios/spool/checkresults
530932 /var/nagios/spool
242624 /var/log
231104 /var/log/apache2
230512 /var/log/apache2/access_log
nagios # ls -al
total 16288
drwxr-xr-x 6 nagios nagios 4096 Aug 16 14:07 .
drwxr-xr-x 15 root root 4096 Aug 15 14:57 ..
-rw-r--r-- 1 root root 0 Oct 25 2010 .keep_net-analyzer_nagios-core-0
drwxr-xr-x 2 nagios nagios 28672 Aug 16 00:00 archives
drwxr-xr-x 4 nagios root 69 Aug 14 17:03 home
-rw-r--r-- 1 nagios nagios 6 Aug 14 12:20 nagios.lock
-rw-r--r-- 1 nagios nagios 5595680 Aug 16 14:07 nagios.log
-rw------- 1 nagios nagios 294912 Nov 9 2009 nagios.tmpQnsGBX
-rw------- 1 nagios nagios 0 Feb 15 2012 nagios.tmpwxgOkC
-rw-r--r-- 1 nagios nagios 2444236 Aug 14 12:20 objects.cache
-rw------- 1 nagios nagios 4178715 Aug 16 13:14 retention.dat
drwsr-sr-x 2 nagios apache 62 Aug 14 12:20 rw
drwxr-xr-x 3 nagios nagios 25 Mar 30 2009 spool
-rw-rw-r-- 1 nagios nagios 4079069 Aug 16 14:07 status.dat
-rw-r--r-- 1 nagios nagios 0 Aug 14 12:20 status.sav
|
Is it safe to delete the whole file from this var directory? AFAICS, the nagios is installed in /
| Code: |
nagios # pwd
/root/nagios
|
|
|
| Back to top |
|
|
mvaterlaus
Apprentice
Joined: 01 Oct 2010
Posts: 234
Location: Switzerland
|
| Posted: Thu Aug 16, 2012 3:00 pm Post subject: |
|
|
i don't know, if you know eclean-dist, but this deletes old installation archives and cleans your portage directory. it is described in [1]. another thing to look at would be your kernel sources under /usr/src. there you can delete every linux-* directory, which you are not using. pay attention when you do this, because you need the sources of the actually running kernel, if you want to rebuild it, and the kernel sources, which you are using for your backup kernel. if you are not sure, which ones you can delete, check the following command, which gives you the actual kernel sources in use:
| Code: |
eselect kernel list
|
eselect is part of the package gentoolkit
[1]http://en.gentoo-wiki.com/wiki/Freeing_Up_Disk_Space |
|
| Back to top |
|
|
midway
Apprentice
Joined: 01 Jun 2012
Posts: 181
Location: somewhr in the blighty
|
| Posted: Thu Aug 16, 2012 3:48 pm Post subject: |
|
|
Okay this eclean-dist has freed only 1.6G. I need to delete loads from the /var dir.
| Code: |
nagios # pwd
/var/nagios
# du -a /var | sort -n -r | head -n 10
41627668 /var
40165284 /var/nagios
39616724 /var/nagios/archives
980656 /var/lib
963064 /var/lib/mysql
530932 /var/nagios/spool/checkresults
530932 /var/nagios/spool
242624 /var/log
231104 /var/log/apache2
230512 /var/log/apache2/access_log
nagios # ls -al
total 16288
drwxr-xr-x 6 nagios nagios 4096 Aug 16 14:07 .
drwxr-xr-x 15 root root 4096 Aug 15 14:57 ..
-rw-r--r-- 1 root root 0 Oct 25 2010 .keep_net-analyzer_nagios-core-0
drwxr-xr-x 2 nagios nagios 28672 Aug 16 00:00 archives
drwxr-xr-x 4 nagios root 69 Aug 14 17:03 home
-rw-r--r-- 1 nagios nagios 6 Aug 14 12:20 nagios.lock
-rw-r--r-- 1 nagios nagios 5595680 Aug 16 14:07 nagios.log
-rw------- 1 nagios nagios 294912 Nov 9 2009 nagios.tmpQnsGBX
-rw------- 1 nagios nagios 0 Feb 15 2012 nagios.tmpwxgOkC
-rw-r--r-- 1 nagios nagios 2444236 Aug 14 12:20 objects.cache
-rw------- 1 nagios nagios 4178715 Aug 16 13:14 retention.dat
drwsr-sr-x 2 nagios apache 62 Aug 14 12:20 rw
drwxr-xr-x 3 nagios nagios 25 Mar 30 2009 spool
-rw-rw-r-- 1 nagios nagios 4079069 Aug 16 14:07 status.dat
-rw-r--r-- 1 nagios nagios 0 Aug 14 12:20 status.sav
|
Is it safe to delete the whole nagios file (above) from this var directory?
AFAICS, the nagios is installed in /
| Code: |
nagios # pwd
/root/nagios
|
Also, here is my kernel output:
| Code: |
src # ls -al
total 8
drwxr-xr-x 3 root root 61 Feb 7 2011 .
drwxr-xr-x 15 root root 4096 Sep 16 2009 ..
-rw-r--r-- 1 root root 0 Apr 14 2007 .keep
lrwxrwxrwx 1 root root 24 Feb 7 2011 linux -> linux-2.6.36-hardened-r9
drwxr-xr-x 25 root root 4096 Feb 7 2011 linux-2.6.36-hardened-r9
src # cd linux
linux # ls -al
total 51372
drwxr-xr-x 25 root root 4096 Feb 7 2011 .
drwxr-xr-x 3 root root 61 Feb 7 2011 ..
-rw-r--r-- 1 root root 927 Feb 7 2011 ..tmp_kallsyms1.o.cmd
-rw-r--r-- 1 root root 927 Feb 7 2011 ..tmp_kallsyms2.o.cmd
-rw-r--r-- 1 root root 664 Feb 7 2011 ..tmp_vmlinux1.cmd
-rw-r--r-- 1 root root 680 Feb 7 2011 ..tmp_vmlinux2.cmd
-rw-r--r-- 1 root root 96525 Feb 7 2011 .config
-rw------- 1 root root 93932 Feb 7 2011 .config.old
-rw-r--r-- 1 root root 936 Oct 20 2010 .gitignore
-rw-r--r-- 1 root root 4021 Oct 20 2010 .mailmap
-rw-r--r-- 1 root root 407 Feb 7 2011 .missing-syscalls.d
-rw-r--r-- 1 root root 1489148 Feb 7 2011 .tmp_System.map
-rw-r--r-- 1 root root 4441478 Feb 7 2011 .tmp_kallsyms1.S
-rw-r--r-- 1 root root 1074792 Feb 7 2011 .tmp_kallsyms1.o
-rw-r--r-- 1 root root 4441478 Feb 7 2011 .tmp_kallsyms2.S
-rw-r--r-- 1 root root 1074792 Feb 7 2011 .tmp_kallsyms2.o
drwxr-xr-x 2 root root 40960 Feb 7 2011 .tmp_versions
-rwxr-xr-x 1 root root 8726818 Feb 7 2011 .tmp_vmlinux1
-rwxr-xr-x 1 root root 9801754 Feb 7 2011 .tmp_vmlinux2
-rw-r--r-- 1 root root 2 Feb 7 2011 .version
-rw-r--r-- 1 root root 668 Feb 7 2011 .vmlinux.cmd
-rw-r--r-- 1 root root 603 Feb 7 2011 .vmlinux.o.cmd
-rw-r--r-- 1 root root 18693 Oct 20 2010 COPYING
-rw-r--r-- 1 root root 94019 Oct 20 2010 CREDITS
drwxr-xr-x 86 root root 8192 Feb 7 2011 Documentation
-rw-r--r-- 1 root root 2440 Oct 20 2010 Kbuild
-rw-r--r-- 1 root root 179296 Oct 20 2010 MAINTAINERS
-rw-r--r-- 1 root root 51851 Feb 7 2011 Makefile
-rw-r--r-- 1 root root 521347 Feb 7 2011 Module.symvers
-rw-r--r-- 1 root root 17459 Oct 20 2010 README
-rw-r--r-- 1 root root 3371 Oct 20 2010 REPORTING-BUGS
-rw-r--r-- 1 root root 1489148 Feb 7 2011 System.map
drwxr-xr-x 27 root root 4096 Feb 7 2011 arch
drwxr-xr-x 2 root root 4096 Feb 7 2011 block
drwxr-xr-x 3 root root 12288 Feb 7 2011 crypto
drwxr-xr-x 89 root root 4096 Feb 7 2011 drivers
drwxr-xr-x 36 root root 4096 Feb 7 2011 firmware
drwxr-xr-x 72 root root 8192 Feb 7 2011 fs
drwxr-xr-x 2 root root 4096 Feb 7 2011 grsecurity
drwxr-xr-x 22 root root 4096 Feb 7 2011 include
drwxr-xr-x 2 root root 4096 Feb 7 2011 init
drwxr-xr-x 2 root root 4096 Feb 7 2011 ipc
drwxr-xr-x 8 root root 12288 Feb 7 2011 kernel
drwxr-xr-x 7 root root 12288 Feb 7 2011 lib
drwxr-xr-x 2 root root 8192 Feb 7 2011 mm
-rw-r--r-- 1 root root 6088 Feb 7 2011 modules.builtin
-rw-r--r-- 1 root root 49440 Feb 7 2011 modules.order
drwxr-xr-x 51 root root 4096 Feb 7 2011 net
drwxr-xr-x 8 root root 132 Feb 7 2011 samples
drwxr-xr-x 13 root root 4096 Feb 7 2011 scripts
drwxr-xr-x 8 root root 4096 Feb 7 2011 security
drwxr-xr-x 21 root root 4096 Feb 7 2011 sound
drwxr-xr-x 5 root root 42 Feb 7 2011 tools
drwxr-xr-x 2 root root 4096 Feb 7 2011 usr
drwxr-xr-x 3 root root 16 Feb 7 2011 virt
-rwxr-xr-x 1 root root 9801754 Feb 7 2011 vmlinux
-rw-r--r-- 1 root root 8830971 Feb 7 2011 vmlinux.o
|
|
|
| Back to top |
|
|
midway
Apprentice
Joined: 01 Jun 2012
Posts: 181
Location: somewhr in the blighty
|
| Posted: Thu Aug 16, 2012 4:34 pm Post subject: |
|
|
okay, I have removed some archived logs of nagios from /var which has freed up almost 21gb. 
thx aCOSwt and mvaterlaus.
Also, any suggestions on the kernels (above)? |
|
| Back to top |
|
|
Ant P.
Watchman
Joined: 18 Apr 2009
Posts: 6920
|
| Posted: Thu Aug 16, 2012 5:20 pm Post subject: |
|
|
| You can run `make distclean` in the kernel dir which deletes all the compiler output, as long as you did `make install` to copy critical files to /boot it won't lose anything important. |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54216
Location: 56N 3W
|
| Posted: Thu Aug 16, 2012 5:34 pm Post subject: |
|
|
nitish.anand,
Broken emerges build up in /var/tmp/portage, I've had 30G in there once. When emerge is not running, remove that entire dir.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
midway
Apprentice
Joined: 01 Jun 2012
Posts: 181
Location: somewhr in the blighty
|
| Posted: Fri Aug 17, 2012 11:04 am Post subject: |
|
|
thx. Ant P. and NeddySeagoon.
Have freed almost another 10gig. Space shows normal now.
| Code: |
~ # du -hx --max-depth=1 /
4.0K /boot
8.4M /bin
0 /dev
15M /etc
195M /lib
0 /mnt
205M /opt
0 /tmp
0 /sys
2.0G /var
3.9G /usr
0 /proc
64M /home
9.0M /sbin
762M /root
12K /.gnupg
0 /run
7.1G /
|
|
|
| Back to top |
|
|
Circuitsoft
Tux's lil' helper
Joined: 14 Jul 2004
Posts: 112
|
| Posted: Fri Aug 17, 2012 10:31 pm Post subject: |
|
|
I have the following line in my /etc/fstab which helps a lot.
| Code: |
none /var/tmp/portage tmpfs defaults 0 0
|
With 6G RAM on this machine, I've never had a problem. |
|
| Back to top |
|
|
|
Kernel & Hardware
