Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[SOLVED] Easy/Simple/Abstruse/Complex bash question
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
sinisterdomestik
l33t
l33t


Joined: 28 Aug 2003
Posts: 685
Location: Texas

PostPosted: Sat Feb 09, 2013 3:31 am    Post subject: [SOLVED] Easy/Simple/Abstruse/Complex bash question Reply with quote

I got this bash script from a thread on here, and I've been looking for days on how to tailor it to my picky needs. The code I have is:
Code:
#!/bin/sh
echo "Note: It is advised to run this script as root"
echo "Please Wait... Creating the list."
buffer="`find / -not \( -path /proc -path /sys -path /dev \) -xdev -printf "%s bytes %p\n" 2>/dev/null`"
echo "Filtered List Created. Now sorting..."
NUM=20
time buffer="`echo "$buffer" | sort -rn | head -$NUM`"
echo "Here is the list of top $NUM biggest files excluding those in /proc, /sys, /dev and mount points :"
echo "$buffer"


I can't, for the life of me, figure out how to get this line to show the file size is "xMB" instead of byte size.
Code:
buffer="`find / -not \( -path /proc -path /sys -path /dev \) -xdev -printf "%s bytes %p\n" 2>/dev/null`"


I know that this is probably a really stupid question, but I've been looking for days ( :( ) to no avail. I also know that once/if someone fixes it, I will feel fairly stupid for not thinking/finding it on my own.

Thank you, from a coding "dummy"
_________________
Thou shalt NEVER speak of removing thine Linux


Last edited by sinisterdomestik on Fri Feb 15, 2013 5:22 pm; edited 1 time in total
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 21489

PostPosted: Sat Feb 09, 2013 4:47 am    Post subject: Reply with quote

Have you considered using du instead? It is better suited to what you want.
Back to top
View user's profile Send private message
sinisterdomestik
l33t
l33t


Joined: 28 Aug 2003
Posts: 685
Location: Texas

PostPosted: Sat Feb 09, 2013 5:24 pm    Post subject: Reply with quote

Hu wrote:
Have you considered using du instead? It is better suited to what you want.


I had been looking into that last night, but can't quite figure out the exact way to write it. Right now, I have my search line as follows
Code:
buffer="'find / -not \( -path /proc -path /sys -patch /dev \) -exec du -ah | sort -rn'"

I know when I run this line by itself
Code:
find / -not \( -path /proc -path /sys -patch /dev \) -exec du -ah | sort -rn
it errors out saying
Quote:
find: missing argument to '-exec'
but I just haven't had the time this morning to search/figure that one out.

Any other tips to get the ball rolling?
_________________
Thou shalt NEVER speak of removing thine Linux
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 21489

PostPosted: Sat Feb 09, 2013 5:51 pm    Post subject: Reply with quote

According to the manual, using du --all --one-file-system --human-readable / will do what you want. For files less than 1M in size, it will be presented in kibibytes instead.
Back to top
View user's profile Send private message
nix213
n00b
n00b


Joined: 08 Feb 2013
Posts: 23
Location: Illinois

PostPosted: Sat Feb 09, 2013 6:13 pm    Post subject: Reply with quote

Quote:

but I just haven't had the time this morning to search/figure that one out.


The error you're getting with find has to do with the "-exec" option usage. You need to use {} to represent the place in the command the filename will be used, and \; to "end" the command you've executed. Eg, if I wanted to edit every text file in a directory, I would use:
find /home -type f -name '*.txt' -exec vim {} \;

I've seen some debian people use '{}' for some reason (it probably depends on how/where it's being used; i've never needed the single quotes though)
_________________
"Wherever you go, go with all your heart." -Confucius
Back to top
View user's profile Send private message
sinisterdomestik
l33t
l33t


Joined: 28 Aug 2003
Posts: 685
Location: Texas

PostPosted: Sat Feb 09, 2013 9:02 pm    Post subject: Reply with quote

Ok, both of those answers helped out. So I'm now making some progress. I have my buffer line now:
Code:
buffer="`find / -not \( -path /proc -path /sys -path /dev \) -exec du -ah --one-file-system /{} \;`"

This runs for a minute, like it should, then just starts throwing out continuous
Code:
du: write error
untill I kill bash manually.

I'm still looking around for different ways to write this and/or see what I'm doing wrong, so maybe I'll find something online, but until then......I shall just cry :'(
_________________
Thou shalt NEVER speak of removing thine Linux
Back to top
View user's profile Send private message
BillWho
Veteran
Veteran


Joined: 03 Mar 2012
Posts: 1600
Location: US

PostPosted: Sat Feb 09, 2013 9:25 pm    Post subject: Reply with quote

sinisterdomestik,

Maybe try filtering out the smaller files since you're looking for top $NUM biggest files :?:

Code:
find / -not \( -path /proc -path /sys -path /dev \) -exec du -xhB1k --apparent-size /{} \;  | awk '$1 >= 50000'

_________________
Good luck :wink:

Since installing gentoo, my life has become one long emerge :)
Back to top
View user's profile Send private message
livibetter
n00b
n00b


Joined: 19 Apr 2009
Posts: 63
Location: Taipei, Taiwan

PostPosted: Sun Feb 10, 2013 3:24 am    Post subject: Reply with quote

sinisterdomestik wrote:
Ok, both of those answers helped out. So I'm now making some progress. I have my buffer line now:
Code:
buffer="`find / -not \( -path /proc -path /sys -path /dev \) -exec du -ah --one-file-system /{} \;`"



You totally misunderstood what Hu's suggested, he meant just run this one command, not with your script:
Code:

$ du --all --one-file-system --human-readable /


If you need skip some files, du has --exclude or --exclude-from options.

If you still want to learn about scripting, then this is what I'd do with your script, untested, written on the fly:

Code:

buffer="`find / -not \( -path /proc -path /sys -path /dev \) -xdev -printf "%s %p\n" 2>/dev/null | while read size path; do echo $((size/1024/1024))MB $path; done`"


You really should use du and forget that script.
Back to top
View user's profile Send private message
sinisterdomestik
l33t
l33t


Joined: 28 Aug 2003
Posts: 685
Location: Texas

PostPosted: Sun Feb 10, 2013 3:34 am    Post subject: Reply with quote

livibetter wrote:
You totally misunderstood what Hu's suggested, he meant just run this one command, not with your script:
Code:

$ du --all --one-file-system --human-readable /


If you need skip some files, du has --exclude or --exclude-from options.


I knew that, I was just trying to add it to the script, so that it would sort the largest X files. By all means, though, if using the command by itself is more efficient, I have no problem with that.

The only reason I was using the script I found in the first place, was that it ran for the whole system, then just listed the 20 largest files so if I needed to make some room on my system, I would know where to start.
_________________
Thou shalt NEVER speak of removing thine Linux
Back to top
View user's profile Send private message
livibetter
n00b
n00b


Joined: 19 Apr 2009
Posts: 63
Location: Taipei, Taiwan

PostPosted: Sun Feb 10, 2013 3:53 am    Post subject: Reply with quote

sinisterdomestik wrote:
livibetter wrote:
You totally misunderstood what Hu's suggested, he meant just run this one command, not with your script:
Code:

$ du --all --one-file-system --human-readable /


If you need skip some files, du has --exclude or --exclude-from options.


I knew that, I was just trying to add it to the script, so that it would sort the largest X files. By all means, though, if using the command by itself is more efficient, I have no problem with that.

The only reason I was using the script I found in the first place, was that it ran for the whole system, then just listed the 20 largest files so if I needed to make some room on my system, I would know where to start.


It is more efficient. And if you need for largest 20, then pipe the output to sort like the script does.

Essentially, what you need is one-liner with pipes, du, sort, and head, you don't need a script like that.
Back to top
View user's profile Send private message
livibetter
n00b
n00b


Joined: 19 Apr 2009
Posts: 63
Location: Taipei, Taiwan

PostPosted: Sun Feb 10, 2013 6:28 am    Post subject: Reply with quote

I should have wrapped up some tests to explain why the original and your modification aren't as efficient as simply using du.

Here are some codes:
Code:
use_du() {
  #   -a   -x                 -h               -S
  du --all --one-file-system --human-readable --separate-dirs |
  #     -h
  sort --human-numeric-sort |
  tail
}

use_find() {
  find -type f -printf '%s %p\n' |
  while read size path; do
    S=$((size/1024/1024))
    if ((S)); then
      echo ${S}MB $path
    else
      S=$((size/1024))
      if ((S)); then
        echo ${S}KB $path
      else
        echo ${size}B $path
      fi
    fi
  done |
  sort --human-numeric-sort |
  tail
}

use_find_simple() {
  find -type f -printf '%s %p\n' |
  while read size path; do
    echo $((size/1024/1024))MB $path
  done |
  sort --human-numeric-sort |
  tail
}

use_find_du() {
  find -type f -exec du --human-readable {} \; |
  sort --human-numeric-sort |
  tail
}


I didn't test on /, but a directory with 3,016 files (`find -type f | wc -l`). If you understand what find's -exec does, you may have thought of that the external command has to be invoked 3,016 times for just using `du` to print out file size in human-readable format for a single file, which is the main reason I called that's inefficient. Even my bash while loop version `use_find` and `use_find_simple` do much quicker.

The detail runtime are, using Bash time builtin function;
Code:
use_du          0.050s
use_find        0.331s
use_find_simple 0.259s
use_find_du     6.236s


There are some notes I have to mention:

du: you can see me adding `-S` option, so the sub-directory's size wouldn't be added up towards parent directory.

And this brings up another note, I add `-type f` to find. As we are only interested in listing files not directories.

Unfortunately, when use `du`, there is no option to skip directories as I can't see any. But it should be fine, you just skip those in the final result on your own or look into those directories because big files are in them.

edit: this might be the best option, no need to print out (and sort by) human-readable numbers and when most of them would be filtered:

Code:
use_find2() {
  find -type f -printf '%s %p\n' |
  sort --numeric-sort |
  tail |
  while read size path; do
    S=$((size/1024/1024))
    if ((S)); then
      echo ${S}MB $path
    else
      S=$((size/1024))
      if ((S)); then
        echo ${S}KB $path
      else
        echo ${size}B $path
      fi
    fi
  done
}
# use_du    0.034s
# use_find2 0.024s
Back to top
View user's profile Send private message
cwr
Veteran
Veteran


Joined: 17 Dec 2005
Posts: 1969

PostPosted: Sun Feb 10, 2013 8:53 am    Post subject: Reply with quote

Generally arithmetic on the command line is done with bc or awk. The easiest
solution to the problem is to add an awk filter to the find command. Try adding:
Code:

 " | awk - F ' ' '{ print $1 / 1000, " K Bytes ",  $3 }'

to the end of the find command. You may have to escape the pipe symbol.
Will
Back to top
View user's profile Send private message
sinisterdomestik
l33t
l33t


Joined: 28 Aug 2003
Posts: 685
Location: Texas

PostPosted: Mon Feb 11, 2013 6:25 am    Post subject: Reply with quote

livibetter wrote:
edit: this might be the best option, no need to print out (and sort by) human-readable numbers and when most of them would be filtered:

Code:
use_find2() {
  find -type f -printf '%s %p\n' |
  sort --numeric-sort |
  tail |
  while read size path; do
    S=$((size/1024/1024))
    if ((S)); then
      echo ${S}MB $path
    else
      S=$((size/1024))
      if ((S)); then
        echo ${S}KB $path
      else
        echo ${size}B $path
      fi
    fi
  done
}
# use_du    0.034s
# use_find2 0.024s


Great post, and that was very useful and helpful, so thank you!! The code above is what I've been messing around with, and so far(at least from what I can tell) have been getting normal results. As far as I can tell, this does what I was looking for.
Code:
#!/bin/sh
find -type f -printf '%s %p\n' |
sort --numeric-sort |
tail |
while read size path; do
   S=$((size/1024/1024))
   if ((S)); then
      echo ${S}MB $path
   else
      S=$((size/1024))
      if ((S)); then
         echo ${S}KB $path
      else
        echo ${size}B $path
      fi
   fi
done
This is what I put in my script, and I put the file in the / directory
Code:
RoosterTooth / # bash find_simple
find: `./proc/12769/task/12769/fd/5': No such file or directory
find: `./proc/12769/task/12769/fdinfo/5': No such file or directory
find: `./proc/12769/fd/5': No such file or directory
find: `./proc/12769/fdinfo/5': No such file or directory
        78MB ./usr/portage/distfiles/linux-3.6.tar.bz2
86MB ./usr/portage/distfiles/firefox-17.0.2esr.source.tar.bz2
136MB ./var/log/Xorg.0.log.old
225MB ./usr/portage/distfiles/qt-everywhere-opensource-src-4.8.4.tar.gz
256MB ./sys/devices/pci0000:00/0000:00:0c.0/0000:02:00.0/resource1
256MB ./sys/devices/pci0000:00/0000:00:0c.0/0000:02:00.0/resource1_wc
698MB ./home/foobar/Movies/********.avi
701MB ./home/foobar/Movies/********.avi
2233MB ./home/foobar/.wine/drive_c/Program Files (x86)/Guild Wars/Gw.dat
134217726MB ./proc/kcore
This is the output from that. The odd part is this:
Quote:
134217726MB ./proc/kcore
I'm not sure what's going on with that, but I'm not TOO worried.

Thanks for all the help. It has been a good learning experience, and I have decided that I should probably have at least ONE bash scripting book around the house lol.
_________________
Thou shalt NEVER speak of removing thine Linux


Last edited by sinisterdomestik on Tue Feb 12, 2013 2:31 am; edited 1 time in total
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 21489

PostPosted: Tue Feb 12, 2013 2:07 am    Post subject: Reply with quote

/proc/kcore is a pseudo-file. You can ignore it. You should probably use the -xdev modifier to instruct find not to traverse into other filesystems.

You can disable the existence of /proc/kcore if you do not need it.
Back to top
View user's profile Send private message
sinisterdomestik
l33t
l33t


Joined: 28 Aug 2003
Posts: 685
Location: Texas

PostPosted: Tue Feb 12, 2013 2:30 am    Post subject: Reply with quote

Hu wrote:
/proc/kcore is a pseudo-file. You can ignore it. You should probably use the -xdev modifier to instruct find not to traverse into other filesystems.

You can disable the existence of /proc/kcore if you do not need it.


Oh perfect!!! That took all the find errors also....beautiful!! Thanks again guys/gals!
_________________
Thou shalt NEVER speak of removing thine Linux
Back to top
View user's profile Send private message
cwr
Veteran
Veteran


Joined: 17 Dec 2005
Posts: 1969

PostPosted: Tue Feb 12, 2013 12:00 pm    Post subject: Reply with quote

You need Machtelt Garrels' "Bash Guide for Beginners" and Mendel Cooper's
"Advanced Bash Scripting Guide" (and the GNU Bash manual) as a start.
They are all available as pdfs - I've never needed anything else.

Will
Back to top
View user's profile Send private message
sinisterdomestik
l33t
l33t


Joined: 28 Aug 2003
Posts: 685
Location: Texas

PostPosted: Tue Feb 12, 2013 1:33 pm    Post subject: Reply with quote

cwr wrote:
You need Machtelt Garrels' "Bash Guide for Beginners" and Mendel Cooper's
"Advanced Bash Scripting Guide" (and the GNU Bash manual) as a start.
They are all available as pdfs - I've never needed anything else.

Will


I was actually looking/reading both of those on tlpd.org at work yesterday, trying to find a few good books to buy/bookmark. So it's good to know I was going in the right direction. Now I have a starting point and can go on, hopefully, figuring things out for myself!
_________________
Thou shalt NEVER speak of removing thine Linux
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum