| View previous topic :: View next topic |
| Author |
Message |
swingkyd
Apprentice
Joined: 12 Jan 2005
Posts: 279
|
 Posted: Mon May 05, 2008 2:24 pm Post subject: [solved]*nix Shell Programming |
 |
|
Hello,
I have a general question about programming using the *nix shell. For example, I have a tonne of CSV files which have lots of data I need to capture. The files are very predictable though:
The last 40 lines of each file, but append the filename to every line as the first field.
Can anyone help me solve this? If not, can anyone help me find a good tutorial where I can learn how to do this kind of stuff? Thanks a tonne!
Here is a sample file:
| Quote: |
DATE OF RUN (Start)= DAY 5 / Month 5 / Year 2008
STARTING TIME= 12: 7: 5:53
RUNID : MZ_001_01.0_100 Frequency Case # 1: 0.0010000 Hz
SCALAR POTENTIALS/TOUCH VOLTAGES/USER DEFINED
|--Profile---|----- Coordinates (meters)------| Touch Voltage (Volts)
Pr.No Point X Y Z Magnitude Ang.(deg) Real Part Imag. Part
====== ====== ====== ======= ======= ======= ======= ======= =======
1, 1, 30.0000000, 0.0000000, 0.0000000, 1.3910098, 0.0000000, 1.3910098, 0.0000000
1, 2, 31.2049999, 0.0000000, 0.0000000, 2.5579591, 0.0000000, 2.5579591, 0.0000000
1, 3, 32.4099998, 0.0000000, 0.0000000, 3.4811647, 0.0000000, 3.4811647, 0.0000000
|
And I would like the output to be streamed to a second file.
Now, I can get all the data without the filename using something like this:
| Code: | #!/bin/bash
sOutFile="./out.csv"
sList=$(ls ../out/MZ_0.001_001_02.0_*.CSV)
:>$sOutFile
echo $sList
for files in $sList; do
# echo $files>> $sOutFile
echo $files
tail -n 40 $files >>$sOutFile
done |
But from here, I don't know how to add the name of the file to each of the 40 lines extracted.
Thanks if you can help
Last edited by swingkyd on Tue May 06, 2008 5:40 pm; edited 1 time in total |
|
| Back to top |
|
 |
spOOwn
Apprentice
Joined: 02 Nov 2002
Posts: 259
Location: Belgium
|
| Posted: Mon May 05, 2008 3:10 pm Post subject: |
|
|
here is something that could perhaps help you, you have to use awk to process each line of your for loop, look at this :
| Code: |
#!/bin/bash
sOutFile="./out.csv"
sList=$(ls ../out/MZ_0.001_001_02.0_*.CSV)
:>$sOutFile
echo $sList
for files in $sList; do
# echo $files>> $sOutFile
echo $files
tail -n 40 $files [b] | awk -v my_var="$sList" '{print $0,my_var}'[/b] >>$sOutFile
done
|
I think that is what you want !
_________________
###########################
# Web site : http://sys-admin.wikidot.com #
########################### |
|
| Back to top |
|
|
swingkyd
Apprentice
Joined: 12 Jan 2005
Posts: 279
|
| Posted: Mon May 05, 2008 3:27 pm Post subject: |
|
|
with an ever-so-small modification, it does what I want. I actually wanted it to pre-pend but append is perfectly fine!
| Code: | #!/bin/bash
sOutFile="./out.csv"
sList=$(ls ../out/MZ_0.001_001_02.0_*.CSV)
:>$sOutFile
echo $sList
for files in $sList; do
# echo $files>> $sOutFile
echo $files
tail -n 40 $files | awk -v my_var="$files" '{print $0,my_var}' >>$sOutFile
done |
Brilliant! and thanks for saving me from pulling my hair out some more.
All I did was use the $files variable instead of $slist. |
|
| Back to top |
|
|
swingkyd
Apprentice
Joined: 12 Jan 2005
Posts: 279
|
| Posted: Mon May 05, 2008 3:30 pm Post subject: |
|
|
| so, just a question, do you know of a good tutorial that can get me proficient at sed and awk? after much "googling" I found lots of examples, man pages etc. but nothing for a newbie like me. I'm wondering if people here have some links handy for help. |
|
| Back to top |
|
|
octanez
Tux's lil' helper
Joined: 17 Apr 2004
Posts: 149
Location: Woods Hole, MA, USA, Earth
|
|
| Back to top |
|
|
spOOwn
Apprentice
Joined: 02 Nov 2002
Posts: 259
Location: Belgium
|
| Posted: Tue May 06, 2008 6:19 am Post subject: |
|
|
if you wanted to pre-pend the output and not append it, change this line :
| Code: |
tail -n 40 $files | awk -v my_var="$sList" '{print $0,my_var}' >>$sOutFile
|
by this one :
| Code: |
tail -n 40 $files | awk -v my_var="$sList" '{print my_var,$0}' >>$sOutFile
|
It change the way awk print the lines you want.
_________________
###########################
# Web site : http://sys-admin.wikidot.com #
########################### |
|
| Back to top |
|
|
swingkyd
Apprentice
Joined: 12 Jan 2005
Posts: 279
|
| Posted: Fri May 16, 2008 9:36 am Post subject: |
|
|
I just thought I'd revisit this code (I've used this already way back when I asked the question) I just thought I'd write out some sort of pseudo code just in case anyone else has no clue like myself 
So from what I understand, awk is a line processor. We are taking the filename and storing it as a variable, which is passed to awk (using -v my_var="$files"). Then each line ($0) is captured and printed to the output with the passed variable appended or prepended using the "print" command.
Cheers and thanks for the quick responses |
|
| Back to top |
|
|
ziggysquatch
Tux's lil' helper
Joined: 16 Nov 2004
Posts: 120
Location: /USA/Minnesota
|
| Posted: Fri May 16, 2008 11:22 am Post subject: |
|
|
| swingkyd wrote: | | so, just a question, do you know of a good tutorial that can get me proficient at sed and awk? after much "googling" I found lots of examples, man pages etc. but nothing for a newbie like me. I'm wondering if people here have some links handy for help. |
I read the Unix Shells by Example by Ellie Quigly and it has loads of info on how to use sed and awk among others. I think its in its fifth edition now or something. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Portage & Programming
