View previous topic :: View next topic |
Author |
Message |
sinisterdomestik l33t
Joined: 28 Aug 2003 Posts: 685 Location: Texas
|
Posted: Sat Feb 09, 2013 3:31 am Post subject: [SOLVED] Easy/Simple/Abstruse/Complex bash question |
|
|
I got this bash script from a thread on here, and I've been looking for days on how to tailor it to my picky needs. The code I have is:
Code: | #!/bin/sh
echo "Note: It is advised to run this script as root"
echo "Please Wait... Creating the list."
buffer="`find / -not \( -path /proc -path /sys -path /dev \) -xdev -printf "%s bytes %p\n" 2>/dev/null`"
echo "Filtered List Created. Now sorting..."
NUM=20
time buffer="`echo "$buffer" | sort -rn | head -$NUM`"
echo "Here is the list of top $NUM biggest files excluding those in /proc, /sys, /dev and mount points :"
echo "$buffer" |
I can't, for the life of me, figure out how to get this line to show the file size is "xMB" instead of byte size.
Code: | buffer="`find / -not \( -path /proc -path /sys -path /dev \) -xdev -printf "%s bytes %p\n" 2>/dev/null`" |
I know that this is probably a really stupid question, but I've been looking for days ( ) to no avail. I also know that once/if someone fixes it, I will feel fairly stupid for not thinking/finding it on my own.
Thank you, from a coding "dummy" _________________ Thou shalt NEVER speak of removing thine Linux
Last edited by sinisterdomestik on Fri Feb 15, 2013 5:22 pm; edited 1 time in total |
|
Back to top |
|
|
Hu Moderator
Joined: 06 Mar 2007 Posts: 21498
|
Posted: Sat Feb 09, 2013 4:47 am Post subject: |
|
|
Have you considered using du instead? It is better suited to what you want. |
|
Back to top |
|
|
sinisterdomestik l33t
Joined: 28 Aug 2003 Posts: 685 Location: Texas
|
Posted: Sat Feb 09, 2013 5:24 pm Post subject: |
|
|
Hu wrote: | Have you considered using du instead? It is better suited to what you want. |
I had been looking into that last night, but can't quite figure out the exact way to write it. Right now, I have my search line as follows
Code: | buffer="'find / -not \( -path /proc -path /sys -patch /dev \) -exec du -ah | sort -rn'" |
I know when I run this line by itself Code: | find / -not \( -path /proc -path /sys -patch /dev \) -exec du -ah | sort -rn | it errors out saying Quote: | find: missing argument to '-exec' | but I just haven't had the time this morning to search/figure that one out.
Any other tips to get the ball rolling? _________________ Thou shalt NEVER speak of removing thine Linux |
|
Back to top |
|
|
Hu Moderator
Joined: 06 Mar 2007 Posts: 21498
|
Posted: Sat Feb 09, 2013 5:51 pm Post subject: |
|
|
According to the manual, using du --all --one-file-system --human-readable / will do what you want. For files less than 1M in size, it will be presented in kibibytes instead. |
|
Back to top |
|
|
nix213 n00b
Joined: 08 Feb 2013 Posts: 23 Location: Illinois
|
Posted: Sat Feb 09, 2013 6:13 pm Post subject: |
|
|
Quote: |
but I just haven't had the time this morning to search/figure that one out.
|
The error you're getting with find has to do with the "-exec" option usage. You need to use {} to represent the place in the command the filename will be used, and \; to "end" the command you've executed. Eg, if I wanted to edit every text file in a directory, I would use:
find /home -type f -name '*.txt' -exec vim {} \;
I've seen some debian people use '{}' for some reason (it probably depends on how/where it's being used; i've never needed the single quotes though) _________________ "Wherever you go, go with all your heart." -Confucius |
|
Back to top |
|
|
sinisterdomestik l33t
Joined: 28 Aug 2003 Posts: 685 Location: Texas
|
Posted: Sat Feb 09, 2013 9:02 pm Post subject: |
|
|
Ok, both of those answers helped out. So I'm now making some progress. I have my buffer line now:
Code: | buffer="`find / -not \( -path /proc -path /sys -path /dev \) -exec du -ah --one-file-system /{} \;`" |
This runs for a minute, like it should, then just starts throwing out continuous untill I kill bash manually.
I'm still looking around for different ways to write this and/or see what I'm doing wrong, so maybe I'll find something online, but until then......I shall just cry :'( _________________ Thou shalt NEVER speak of removing thine Linux |
|
Back to top |
|
|
BillWho Veteran
Joined: 03 Mar 2012 Posts: 1600 Location: US
|
Posted: Sat Feb 09, 2013 9:25 pm Post subject: |
|
|
sinisterdomestik,
Maybe try filtering out the smaller files since you're looking for top $NUM biggest files
Code: | find / -not \( -path /proc -path /sys -path /dev \) -exec du -xhB1k --apparent-size /{} \; | awk '$1 >= 50000' |
_________________ Good luck
Since installing gentoo, my life has become one long emerge |
|
Back to top |
|
|
livibetter n00b
Joined: 19 Apr 2009 Posts: 63 Location: Taipei, Taiwan
|
Posted: Sun Feb 10, 2013 3:24 am Post subject: |
|
|
sinisterdomestik wrote: | Ok, both of those answers helped out. So I'm now making some progress. I have my buffer line now:
Code: | buffer="`find / -not \( -path /proc -path /sys -path /dev \) -exec du -ah --one-file-system /{} \;`" |
|
You totally misunderstood what Hu's suggested, he meant just run this one command, not with your script:
Code: |
$ du --all --one-file-system --human-readable /
|
If you need skip some files, du has --exclude or --exclude-from options.
If you still want to learn about scripting, then this is what I'd do with your script, untested, written on the fly:
Code: |
buffer="`find / -not \( -path /proc -path /sys -path /dev \) -xdev -printf "%s %p\n" 2>/dev/null | while read size path; do echo $((size/1024/1024))MB $path; done`"
|
You really should use du and forget that script. |
|
Back to top |
|
|
sinisterdomestik l33t
Joined: 28 Aug 2003 Posts: 685 Location: Texas
|
Posted: Sun Feb 10, 2013 3:34 am Post subject: |
|
|
livibetter wrote: | You totally misunderstood what Hu's suggested, he meant just run this one command, not with your script:
Code: |
$ du --all --one-file-system --human-readable /
|
If you need skip some files, du has --exclude or --exclude-from options. |
I knew that, I was just trying to add it to the script, so that it would sort the largest X files. By all means, though, if using the command by itself is more efficient, I have no problem with that.
The only reason I was using the script I found in the first place, was that it ran for the whole system, then just listed the 20 largest files so if I needed to make some room on my system, I would know where to start. _________________ Thou shalt NEVER speak of removing thine Linux |
|
Back to top |
|
|
livibetter n00b
Joined: 19 Apr 2009 Posts: 63 Location: Taipei, Taiwan
|
Posted: Sun Feb 10, 2013 3:53 am Post subject: |
|
|
sinisterdomestik wrote: | livibetter wrote: | You totally misunderstood what Hu's suggested, he meant just run this one command, not with your script:
Code: |
$ du --all --one-file-system --human-readable /
|
If you need skip some files, du has --exclude or --exclude-from options. |
I knew that, I was just trying to add it to the script, so that it would sort the largest X files. By all means, though, if using the command by itself is more efficient, I have no problem with that.
The only reason I was using the script I found in the first place, was that it ran for the whole system, then just listed the 20 largest files so if I needed to make some room on my system, I would know where to start. |
It is more efficient. And if you need for largest 20, then pipe the output to sort like the script does.
Essentially, what you need is one-liner with pipes, du, sort, and head, you don't need a script like that. |
|
Back to top |
|
|
livibetter n00b
Joined: 19 Apr 2009 Posts: 63 Location: Taipei, Taiwan
|
Posted: Sun Feb 10, 2013 6:28 am Post subject: |
|
|
I should have wrapped up some tests to explain why the original and your modification aren't as efficient as simply using du.
Here are some codes:
Code: | use_du() {
# -a -x -h -S
du --all --one-file-system --human-readable --separate-dirs |
# -h
sort --human-numeric-sort |
tail
}
use_find() {
find -type f -printf '%s %p\n' |
while read size path; do
S=$((size/1024/1024))
if ((S)); then
echo ${S}MB $path
else
S=$((size/1024))
if ((S)); then
echo ${S}KB $path
else
echo ${size}B $path
fi
fi
done |
sort --human-numeric-sort |
tail
}
use_find_simple() {
find -type f -printf '%s %p\n' |
while read size path; do
echo $((size/1024/1024))MB $path
done |
sort --human-numeric-sort |
tail
}
use_find_du() {
find -type f -exec du --human-readable {} \; |
sort --human-numeric-sort |
tail
}
|
I didn't test on /, but a directory with 3,016 files (`find -type f | wc -l`). If you understand what find's -exec does, you may have thought of that the external command has to be invoked 3,016 times for just using `du` to print out file size in human-readable format for a single file, which is the main reason I called that's inefficient. Even my bash while loop version `use_find` and `use_find_simple` do much quicker.
The detail runtime are, using Bash time builtin function;
Code: | use_du 0.050s
use_find 0.331s
use_find_simple 0.259s
use_find_du 6.236s
|
There are some notes I have to mention:
du: you can see me adding `-S` option, so the sub-directory's size wouldn't be added up towards parent directory.
And this brings up another note, I add `-type f` to find. As we are only interested in listing files not directories.
Unfortunately, when use `du`, there is no option to skip directories as I can't see any. But it should be fine, you just skip those in the final result on your own or look into those directories because big files are in them.
edit: this might be the best option, no need to print out (and sort by) human-readable numbers and when most of them would be filtered:
Code: | use_find2() {
find -type f -printf '%s %p\n' |
sort --numeric-sort |
tail |
while read size path; do
S=$((size/1024/1024))
if ((S)); then
echo ${S}MB $path
else
S=$((size/1024))
if ((S)); then
echo ${S}KB $path
else
echo ${size}B $path
fi
fi
done
}
# use_du 0.034s
# use_find2 0.024s
|
|
|
Back to top |
|
|
cwr Veteran
Joined: 17 Dec 2005 Posts: 1969
|
Posted: Sun Feb 10, 2013 8:53 am Post subject: |
|
|
Generally arithmetic on the command line is done with bc or awk. The easiest
solution to the problem is to add an awk filter to the find command. Try adding:
Code: |
" | awk - F ' ' '{ print $1 / 1000, " K Bytes ", $3 }'
|
to the end of the find command. You may have to escape the pipe symbol.
Will |
|
Back to top |
|
|
sinisterdomestik l33t
Joined: 28 Aug 2003 Posts: 685 Location: Texas
|
Posted: Mon Feb 11, 2013 6:25 am Post subject: |
|
|
livibetter wrote: | edit: this might be the best option, no need to print out (and sort by) human-readable numbers and when most of them would be filtered:
Code: | use_find2() {
find -type f -printf '%s %p\n' |
sort --numeric-sort |
tail |
while read size path; do
S=$((size/1024/1024))
if ((S)); then
echo ${S}MB $path
else
S=$((size/1024))
if ((S)); then
echo ${S}KB $path
else
echo ${size}B $path
fi
fi
done
}
# use_du 0.034s
# use_find2 0.024s
|
|
Great post, and that was very useful and helpful, so thank you!! The code above is what I've been messing around with, and so far(at least from what I can tell) have been getting normal results. As far as I can tell, this does what I was looking for.
Code: | #!/bin/sh
find -type f -printf '%s %p\n' |
sort --numeric-sort |
tail |
while read size path; do
S=$((size/1024/1024))
if ((S)); then
echo ${S}MB $path
else
S=$((size/1024))
if ((S)); then
echo ${S}KB $path
else
echo ${size}B $path
fi
fi
done | This is what I put in my script, and I put the file in the / directory
Code: | RoosterTooth / # bash find_simple
find: `./proc/12769/task/12769/fd/5': No such file or directory
find: `./proc/12769/task/12769/fdinfo/5': No such file or directory
find: `./proc/12769/fd/5': No such file or directory
find: `./proc/12769/fdinfo/5': No such file or directory
78MB ./usr/portage/distfiles/linux-3.6.tar.bz2
86MB ./usr/portage/distfiles/firefox-17.0.2esr.source.tar.bz2
136MB ./var/log/Xorg.0.log.old
225MB ./usr/portage/distfiles/qt-everywhere-opensource-src-4.8.4.tar.gz
256MB ./sys/devices/pci0000:00/0000:00:0c.0/0000:02:00.0/resource1
256MB ./sys/devices/pci0000:00/0000:00:0c.0/0000:02:00.0/resource1_wc
698MB ./home/foobar/Movies/********.avi
701MB ./home/foobar/Movies/********.avi
2233MB ./home/foobar/.wine/drive_c/Program Files (x86)/Guild Wars/Gw.dat
134217726MB ./proc/kcore | This is the output from that. The odd part is this: Quote: | 134217726MB ./proc/kcore | I'm not sure what's going on with that, but I'm not TOO worried.
Thanks for all the help. It has been a good learning experience, and I have decided that I should probably have at least ONE bash scripting book around the house lol. _________________ Thou shalt NEVER speak of removing thine Linux
Last edited by sinisterdomestik on Tue Feb 12, 2013 2:31 am; edited 1 time in total |
|
Back to top |
|
|
Hu Moderator
Joined: 06 Mar 2007 Posts: 21498
|
Posted: Tue Feb 12, 2013 2:07 am Post subject: |
|
|
/proc/kcore is a pseudo-file. You can ignore it. You should probably use the -xdev modifier to instruct find not to traverse into other filesystems.
You can disable the existence of /proc/kcore if you do not need it. |
|
Back to top |
|
|
sinisterdomestik l33t
Joined: 28 Aug 2003 Posts: 685 Location: Texas
|
Posted: Tue Feb 12, 2013 2:30 am Post subject: |
|
|
Hu wrote: | /proc/kcore is a pseudo-file. You can ignore it. You should probably use the -xdev modifier to instruct find not to traverse into other filesystems.
You can disable the existence of /proc/kcore if you do not need it. |
Oh perfect!!! That took all the find errors also....beautiful!! Thanks again guys/gals! _________________ Thou shalt NEVER speak of removing thine Linux |
|
Back to top |
|
|
cwr Veteran
Joined: 17 Dec 2005 Posts: 1969
|
Posted: Tue Feb 12, 2013 12:00 pm Post subject: |
|
|
You need Machtelt Garrels' "Bash Guide for Beginners" and Mendel Cooper's
"Advanced Bash Scripting Guide" (and the GNU Bash manual) as a start.
They are all available as pdfs - I've never needed anything else.
Will |
|
Back to top |
|
|
sinisterdomestik l33t
Joined: 28 Aug 2003 Posts: 685 Location: Texas
|
Posted: Tue Feb 12, 2013 1:33 pm Post subject: |
|
|
cwr wrote: | You need Machtelt Garrels' "Bash Guide for Beginners" and Mendel Cooper's
"Advanced Bash Scripting Guide" (and the GNU Bash manual) as a start.
They are all available as pdfs - I've never needed anything else.
Will |
I was actually looking/reading both of those on tlpd.org at work yesterday, trying to find a few good books to buy/bookmark. So it's good to know I was going in the right direction. Now I have a starting point and can go on, hopefully, figuring things out for myself! _________________ Thou shalt NEVER speak of removing thine Linux |
|
Back to top |
|
|
|