| View previous topic :: View next topic |
| Author |
Message |
efrank
n00b
Joined: 12 Mar 2021
Posts: 9
Location: Caracas
|
 Posted: Fri Aug 12, 2022 12:30 pm Post subject: /dev/root is full. eclean didn't help |
 |
|
Hi guys.
I have a file server with netatalk and samba services, no mysql or any other database. It has one single partition where all system resides.
Turns out that the single partition is full. I've read that one quick way to solve this would be to use eclean, so I issued both | Code: | | eclean --deep distfiles |
and | Code: | | eclean --deep packages. |
It didn't help much, because the partition still has no space left. Even after a reboot things haven't changed.
df -h:
| Code: | Filesystem Size Used Avail Use% Mounted on
/dev/root 362G 361G 0 100% /
devtmpfs 10M 0 10M 0% /dev
tmpfs 401M 1.5M 399M 1% /run
cgroup_root 10M 0 10M 0% /sys/fs/cgroup
shm 1001M 0 1001M 0% /dev/shm
/dev/sdb1 917G 744G 127G 86% /mnt/trabajos |
du -hsx /* | sort -rh | head -n 40:
| Code: | 357G /mnt
2.8G /usr
1.6G /var
25M /boot
21M /lib
15M /etc
9.6M /bin
8.1M /sbin
1.5M /run
32K /root
32K /home
20K /tmp
16K /lost+found
4.0K /opt
4.0K /media
0 /sys
0 /proc
0 /dev |
How can I resolve this? I'm at the end of my rope for now. Any hints or help are highly appreciated. The current situation prevents me from maintaining the server; can't emerge anything.
Thanks in advance! |
|
| Back to top |
|
 |
freke
l33t
Joined: 23 Jan 2003
Posts: 977
Location: Somewhere in Denmark
|
| Posted: Fri Aug 12, 2022 12:37 pm Post subject: |
|
|
| I would take a look at contents of /mnt - do you expect that much data there? (ignoring /mnt/trabajos which is mounted on its own partition) |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3342
Location: Rasi, Finland
|
| Posted: Fri Aug 12, 2022 12:50 pm Post subject: |
|
|
| freke wrote: | | I would take a look at contents of /mnt - do you expect that much data there? (ignoring /mnt/trabajos which is mounted on its own partition) |
... or under /mnt/trabajos mount.
I suspect something is writing data there without checking if it is a mount point.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
efrank
n00b
Joined: 12 Mar 2021
Posts: 9
Location: Caracas
|
| Posted: Fri Aug 12, 2022 2:13 pm Post subject: |
|
|
| freke wrote: | | I would take a look at contents of /mnt - do you expect that much data there? (ignoring /mnt/trabajos which is mounted on its own partition) |
No, I don't expect that much data there. On the other hand, I've looked inside of /mnt, but there is not much to see, except for a couple of mount points.
How would you track what is writing to it? |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3342
Location: Rasi, Finland
|
| Posted: Fri Aug 12, 2022 2:14 pm Post subject: |
|
|
Unmount /mnt/trabajos. Then see if there are still files under it.
Or make another "mnt" directory and bind mount /mnt into it: | as root: | mkdir /mnt2
mount -o bind /mnt /mnt2
ls -lhaF /mnt2/ |
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
efrank
n00b
Joined: 12 Mar 2021
Posts: 9
Location: Caracas
|
| Posted: Fri Aug 12, 2022 2:20 pm Post subject: |
|
|
| Zucca wrote: | | freke wrote: | | I would take a look at contents of /mnt - do you expect that much data there? (ignoring /mnt/trabajos which is mounted on its own partition) |
... or under /mnt/trabajos mount.
I suspect something is writing data there without checking if it is a mount point. |
This is a list of /mnt:
| Code: | drwxr-xr-x 4 root root 4.0K Aug 22 2021 .
drwxr-xr-x 20 root root 4.0K Aug 12 04:50 ..
-rw-r--r-- 1 root root 0 Aug 16 2021 .keep
drwxrwxrwx 1780 nobody nobody 68K Aug 11 16:51 respaldo
drwxrwxrwx 1781 nobody nobody 72K Aug 12 09:59 trabajos |
|
|
| Back to top |
|
|
efrank
n00b
Joined: 12 Mar 2021
Posts: 9
Location: Caracas
|
| Posted: Fri Aug 12, 2022 2:24 pm Post subject: |
|
|
| Zucca wrote: | Unmount /mnt/trabajos. Then see if there are still files under it.
Or make another "mnt" directory and bind mount /mnt into it: | as root: | mkdir /mnt2
mount -o bind /mnt /mnt2
ls -lhaF /mnt2/ |
|
ls -lhaF /mnt2/:
| Code: | total 84K
drwxr-xr-x 4 root root 4.0K Aug 22 2021 ./
drwxr-xr-x 21 root root 4.0K Aug 12 10:24 ../
-rw-r--r-- 1 root root 0 Aug 16 2021 .keep
drwxrwxrwx 1780 nobody nobody 68K Aug 11 16:51 respaldo/
drwxr-xr-x 2 root root 4.0K Aug 22 2021 trabajos/ |
|
|
| Back to top |
|
|
efrank
n00b
Joined: 12 Mar 2021
Posts: 9
Location: Caracas
|
| Posted: Fri Aug 12, 2022 2:30 pm Post subject: |
|
|
du -h -d1 /mnt2:
| Code: | 4.0K /mnt2/trabajos
357G /mnt2/respaldo
357G /mnt2 |
the problem seems to be /mnt2/respaldo. |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3342
Location: Rasi, Finland
|
| Posted: Fri Aug 12, 2022 2:35 pm Post subject: |
|
|
| efrank wrote: | du -h -d1 /mnt2:
| Code: | 4.0K /mnt2/trabajos
357G /mnt2/respaldo
357G /mnt2 |
the problem seems to be /mnt2/respaldo. |
Yes.
You can now safely unmount /mnt2. Those files should be under /mnt/respaldo too if there is nothing mounted over it.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
efrank
n00b
Joined: 12 Mar 2021
Posts: 9
Location: Caracas
|
| Posted: Fri Aug 12, 2022 4:49 pm Post subject: [SOLVED] /dev/root is full. eclean didn't help |
|
|
Got rid of /mnt/respaldo/ and things got back to normal; /dev/root is not full anymore.
Thanks a lot Zucca and freke for the input that let me solved this. |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3342
Location: Rasi, Finland
|
| Posted: Fri Aug 12, 2022 5:32 pm Post subject: |
|
|
Clean up by... if you still have that directory there. ;)
Should you have some drive mounted to /mnt/respaldo? As a backup drive?
If yes, then your backup program most probably backs your files there. If the drive isn't mounted there then it back your files into your root partition. And that's how your root partition fills up.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
Goverp
Advocate
Joined: 07 Mar 2007
Posts: 2003
|
| Posted: Fri Aug 12, 2022 6:33 pm Post subject: |
|
|
| Zucca wrote: | Should you have some drive mounted to /mnt/respaldo? As a backup drive?
If yes, then your backup program most probably backs your files there. If the drive isn't mounted there then it back your files into your root partition. And that's how your root partition fills up. |
More importantly, if you recovered the space by deleting it, you no longer have that backup...
_________________
Greybeard |
|
| Back to top |
|
|
szatox
Advocate
Joined: 27 Aug 2013
Posts: 3133
|
| Posted: Fri Aug 12, 2022 9:26 pm Post subject: |
|
|
| Quote: | | If yes, then your backup program most probably backs your files there. If the drive isn't mounted there then it back your files into your root partition. And that's how your root partition fills up. |
Which begs a question: How do we prevent such a mistake?
Pro tip: set the permissions on an inactive mountpoint to 000, and ensure the application is NOT running as root.
When the mountpoint is active, it hides both the original directory AND its permissions.
This way all access works fine when some other FS is mounted on top of your target path, but should that FS go missing, all write attempts would fail immediately with a loud bang, which gives you a chance to notice and correct the issue. |
|
| Back to top |
|
|
Hu
Moderator
Joined: 06 Mar 2007
Posts: 21619
|
| Posted: Fri Aug 12, 2022 10:08 pm Post subject: |
|
|
| For some filesystems, you can go one better than that. After making the permissions change szatox recommends, also make that directory immutable. Then even root cannot create files there. As with what szatox recommends, once the mount is in place, all properties of the locked directory are hidden, so the immutability will not prevent backups when the mount is present. |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3342
Location: Rasi, Finland
|
| Posted: Sat Aug 13, 2022 7:41 am Post subject: |
|
|
| Hu wrote: | | After making the permissions change szatox recommends, also make that directory immutable. |
Now I'm tempted to create some init script which makes all the (empty) mount points immutable... ;)
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
|
Other Things Gentoo
