Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
mrtg script question [learning]
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Gentoo Chat
View previous topic :: View next topic  
Author Message
jonfr
Veteran
Veteran


Joined: 20 Jul 2003
Posts: 1008
Location: Denmark

PostPosted: Fri Mar 15, 2013 12:07 am    Post subject: mrtg script question [learning] Reply with quote

I am trying to get an working counter for an earthquake web page that I need to monitor. This script is supposed to supply the data for mrtg to count the reviewed earthquakes. But nothing happens and everything remains at zero.

The script in question is like this.

Code:
#earthquake count shell V0.0.1
#!/bin/bash

URL="http://hraun.vedur.is/ja/viku/2013/vika_11/listi"
#TODAYSDATE=`date +%Y-%m-%d`
#TODAYSDATE=`date +%Y%m%d`
TODAYSDATE=`date +%Y%m%d`

COUNT=`lynx -dump ${URL} | grep ${TODAYSDATE} | grep -vi updated | wc -l`
echo ${COUNT}
echo ${COUNT}
#echo ${COUNT}
echo "uptime"
echo "hostname"

exit 0


I need to be able to move from week 11 to week 12 (and so on) once it happens. Since it does not happen on it's own. If this does not work. I just have to cancel this.

Thanks for the help.


Last edited by jonfr on Sun Mar 17, 2013 9:04 pm; edited 1 time in total
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6227
Location: Room 101

PostPosted: Fri Mar 15, 2013 3:36 pm    Post subject: Reply with quote

jonfr ...

I think lynx is dumping the file and not sending to stdout ...

Code:
# wget --quiet -O - http://hraun.vedur.is/ja/viku/2013/vika_11/listi | grep $(date +%Y%m%d)
  87 20130315 044740.900 63.99043 -20.70260  7.942  0.49  0.14
  88 20130315 065642.904 65.02353 -16.67575  3.703  1.61  1.21

Also, if this is the expected output there doesn't seem to be the necessity to 'grep -v updated'

Code:
# wget --quiet -O - http://hraun.vedur.is/ja/viku/2013/vika_11/listi | grep $(date +%Y%m%d) | wc -l
2

HTH & best ... khay
Back to top
View user's profile Send private message
jonfr
Veteran
Veteran


Joined: 20 Jul 2003
Posts: 1008
Location: Denmark

PostPosted: Fri Mar 15, 2013 6:53 pm    Post subject: Reply with quote

This script, as is shown here seems to work. The only problem remains is that I now have to figure out a way to get the latest data from this web site. As the "vika_11" changes to "vika_12" next week. Since this earthquake data is divided by weeks. Not months.
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6227
Location: Room 101

PostPosted: Fri Mar 15, 2013 8:07 pm    Post subject: Reply with quote

johfr ...

well, assuming 54 weeks

Code:
# for ((i=11; i<55; i++)) ; do echo "count to $i" ; done

so, similarly with the above, some math to increment the week:

Code:
#!/bin/bash
#earthquake count shell V0.0.1

WEEK="$(cat /path/to/increment.txt)"
URL="http://hraun.vedur.is/ja/viku/2013/vika_${WEEK}/listi"
TODAYSDATE="$(date +%Y%m%d)"
COUNT="$(lynx -dump ${URL} | grep ${TODAYSDATE} | wc -l)"

echo "$(( ${WEEK} + 1 ))" > /path/to/increment.txt
echo "${COUNT}"
echo "${COUNT}"
echo "uptime"
echo "hostname"

best ... khay
Back to top
View user's profile Send private message
jonfr
Veteran
Veteran


Joined: 20 Jul 2003
Posts: 1008
Location: Denmark

PostPosted: Fri Mar 15, 2013 10:26 pm    Post subject: Reply with quote

So this dumps the data into a file locally? I am still learning basic scripting on bash shell. Can I use ${YEAR} since this moves from 2013 to 2014 when the year changes.

Thanks for the help :)
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6227
Location: Room 101

PostPosted: Sat Mar 16, 2013 7:20 am    Post subject: Reply with quote

jonfr wrote:
So this dumps the data into a file locally? I am still learning basic scripting on bash shell. Can I use ${YEAR} since this moves from 2013 to 2014 when the year changes.

jonfr ... no, all it does is increment the week, the previous week (number) having been stored in a file (increment.txt) is read in the next time the script is run, then the value for ${WEEK} is incremented by one and the old increment.txt is updated/over-written with the new value. It was in answer to "[t]he only problem remains is that I now have to figure out a way to get the latest data from this web site. As the "vika_11" changes to "vika_12" next week."

If you need to write data to a file for future use you can do something like the following:

Code:
echo "count=${COUNT}" >> /path/to/quake_${TODAYSDATE}.log

Otherwise the 'echo' goes to 'stdout' (standard output).

The script is a little confusing to me as I don't know what 'hostname', or 'uptime', have to do with any of this.

jonfr wrote:
Thanks for the help

You're welcome & best ... khay
Back to top
View user's profile Send private message
jonfr
Veteran
Veteran


Joined: 20 Jul 2003
Posts: 1008
Location: Denmark

PostPosted: Sat Mar 16, 2013 3:17 pm    Post subject: Reply with quote

I guess the hostname and uptime lines are not needed. If I run the script from shell. I get this output.

Code:
./eq.imo.rev.sh
19
19
uptime
hostname


Someone did write this script for me a while back. I have been doing my best to figure out the shell script since then when time allows for it.
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6227
Location: Room 101

PostPosted: Sat Mar 16, 2013 4:26 pm    Post subject: Reply with quote

jonfr wrote:
Someone did write this script for me a while back. I have been doing my best to figure out the shell script since then when time allows for it.

jonfr ... ok, I guess what your looking for is something like the following:

Code:
#!/bin/bash
# earthquake count shell v0.0.2

WEEK="$(cat /path/to/increment.txt)"
URL="http://hraun.vedur.is/ja/viku/2013/vika_${WEEK}/listi"
DATE="$(date +%Y%m%d)"
DATA="$(wget --quiet -O - ${URL} | grep ${DATE} | wc -l)"

echo "$((WEEK+1))" > /path/to/increment.txt
echo "${DATE}=${DATA}" >> /path/to/file.log

You would then create an 'increment.txt' and insert in the the number of the current week, ie 11, make sure the '/path/to/' for the files exist, and run the script. This will create two files 'file.log' which is the data formated like so:

Code:
20130316=2
20130323=8
20130330=5
....

and 'increment.txt' which will be a number, in this case '12' (as its this week, 11, plus 1). Each week the script is run the incriment will be adjusted for the following weeks run, and data added to the log. All you need then do is make sure the script is run at a regular weekly interval.

best ... khay
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Sun Mar 17, 2013 2:43 pm    Post subject: Reply with quote

khayyam wrote:
Code:
WEEK="$(cat /path/to/increment.txt)"

uuoc. Plus you should use a variable set at top of script, or passed in as a parameter for the filename (jonfr: note that we don't use all-caps names for general variables):
Code:
read -r week < "$inc_file" || week=$(date ..)

It's always worth getting beginners used to both filename parameters, which must be quoted, and to reading lines from a file. Globs are fundamental too: the shell handles all filenames correctly: so long as we quote them on use, so can we.
Code:
for f in "$dir/"*; do
   [[ -f $f ]] || continue
   while read -r line; do
     echo "$line"
   done < "$f"
done

[[ .. ]] doesn't field-split in bash, so most variables therein don't need to be quoted: however it will glob-match the right-hand side of an =, so
Code:
[[ $foo = "$bar" ]]
is an exact match, and
Code:
[[ $foo = $bar ]]
uses $bar as a glob.
Code:
[[ $foo = *"$bar"* ]]
is more common, and tests whether $bar is a substring of $foo.

A similar situation applies to the RHS of =~ (for regex-matching.) If a variable is quoted therein, it's treated a literal text. This behaviour is a change between bash-3.1 and bash-3.2+ for consistency with the above. The most compatible way to use regexen in bash is with a variable, eg:
Code:
readonly RE='((foo|bar|baz)[0-9]+_)+lala'
foo() {
   if [[ $1 =~ $RE ]]; then
   .. # if needed, use $BASH_REMATCH or ${BASH_REMATCH[n]}
   fi

This will work on any bash with =~ (3.1 and above iirc.)
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6227
Location: Room 101

PostPosted: Sun Mar 17, 2013 4:54 pm    Post subject: Reply with quote

steveL wrote:
uuoc. Plus you should use a variable set at top of script, or passed in as a parameter for the filename

steveL ... yes, uuoc ... I would normally use 'read', but I didn't want to confuse an obvious pre-beginner with details. I tried to keep the script simple, and close to the original, as it was clear that even that, which was very basic, wasn't totally understood. I stop short at 'cat input|' ;) ... anyhow, my thought was that even those with minimal exposure to the shell tend to have been exposed to 'cat' and so I hoped what it was doing would be understood.

steveL wrote:
Code:
for f in "$dir/"*; do
   [[ -f $f ]] || continue
   while read -r line; do
     echo "$line"
   done < "$f"
done

I think you will have lost them at this stage ...

best ... khay
Back to top
View user's profile Send private message
jonfr
Veteran
Veteran


Joined: 20 Jul 2003
Posts: 1008
Location: Denmark

PostPosted: Sun Mar 17, 2013 9:04 pm    Post subject: Reply with quote

Thanks for the help.

bash script is just what I know for now. But I think mrtg can run anything in this terms. But I do not know anything as if yet. The problem so far is that this script also needs to move from one year to the next (2013 to 2014 for instance). I am still looking into this if this can actually be done. I am thankful for the help. Since scripts are bit out of what I know in Gentoo Linux.
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6227
Location: Room 101

PostPosted: Sun Mar 17, 2013 9:28 pm    Post subject: Reply with quote

jonfr wrote:
The problem so far is that this script also needs to move from one year to the next (2013 to 2014 for instance). I am still looking into this if this can actually be done.

jonfr ...

Its quite simple really, 'date' can be used to output all kinds of formats:

Code:
# date +%G
2013

So, the above could be modified to use the year, and, iso week number, like so:

Code:
url="http://hraun.vedur.is/ja/viku/$(date +%G)/vika_$(date +%g)/listi"

Infact, I should have made that suggestion previously, rather than use a file to store the increment ... what was I thinking :P

Code:
#!/bin/bash
# earthquake count shell v0.0.3

url="http://hraun.vedur.is/ja/viku/$(date +%G)/vika_$(date +%g)/listi"
data="$(wget --quiet -O - ${url} | grep $(date +%Y%m%d) | wc -l)"

echo "${data}" >> /path/to/file.log

best ... khay
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Mon Mar 18, 2013 12:49 pm    Post subject: Reply with quote

khayyam wrote:
steveL ... yes, uuoc ... I would normally use 'read', but I didn't want to confuse an obvious pre-beginner with details.
So instead you reinforce a bad habit, just when they're starting out?
When I was starting out, I really appreciated ##bash (as it was then) simply telling me the efficient idioms, so I could learn how to do things correctly from the beginning.
Quote:
I tried to keep the script simple, and close to the original, as it was clear that even that, which was very basic, wasn't totally understood. I stop short at 'cat input|' ;) ... anyhow, my thought was that even those with minimal exposure to the shell tend to have been exposed to 'cat' and so I hoped what it was doing would be understood.

Yeah and that's just unnecessary: all you end up doing is giving the impression that that's what shellscripters would do, and it blatantly is not. Better just to simply explain what you're doing and why you're doing it, if it isn't obvious.

The latter can be hard to differentiate, so I'll explain what I was doing ;)
Code:
for f in "$dir/"*; do

..is a glob or wildcard pattern. The for makes f step through the values from the glob. As explained above the shell will handle any filename correctly, so long as we quote it on use. Note that we can use many globs, and combine with say a bash array:
Code:
for f in "$dir/"*.conf "$HOME/.$prog/"*.conf "${config[@]}"; do

Also, many beginners (and even people who've been scripting for years) don't know that
Code:
for i; do

..steps through the argv array "$@", ie the function or shellscript parameters. If there are none, then the loop is never entered.
Code:
   [[ -f $f ]] || continue

Here we test whether the file exists, and is a regular file we can read. This is important as *.foo will expand to plain '*.foo' if no files match. If there is not a matching file (it could be a directory) then we continue and try the next one, or exit the loop if there are no more.

As explained before, we don't need to quote most expansions inside [[..]], though we usually need to quote variables on the RHS of '='. Either way it won't cause a syntax error like it would in [..], since no field/word-splitting goes on in bash [[..]].
If you want to see what types of things you can check, use help test at the terminal. The predicates that apply to test also apply to [[..]] (help [[ shows how they can be combined with standard C-type booleans.)
Code:
   while read -r line; do
     echo "$line"
   done < "$f"

This is the canonical example of how to read lines from a file, and do something with them. The redirection < "$f" means that stdin comes from the file, for the whole of the compound command (while) that it applies to. It's important to use read -r since otherwise backslashes in the input are interpreted. Note that read -r var will strip trailing and leading spaces from the line. If you want the whole line use:
Code:
   while read -r; do
     echo "$REPLY"
   done < "$f"

You can also read more than one field, eg:
Code:
while read -r date _ url _; do
   echo "$date $url"
done < "$f" | sort -k1,1 > "$o"

There '_' is a junk variable that soaks up the last field (and any we don't want); one could equally use the word 'junk' but the idiom is to use _ since it is set after every command by bash, and this way reads better.
Quote:
I think you will have lost them at this stage ...

Only if it's not explained. I had thought you might take up the baton, since what's there is so fundamental to shell-scripting, or that it might inspire someone to ask what was going on, either here or in #bash on IRC:chat.freenode.net. But yeah, I have a tendency just to correct things, and leave it at that til, especially when I'm busy or tired (I was both ;)
Code:
data="$(wget --quiet -O - ${url} | grep $(date +%Y%m%d) | wc -l)"
 
echo "${data}" >> /path/to/file.log

could simply be:
Code:
wget --quiet -O - "$url" | grep "$(date +%Y%m%d)" | wc -l >> "$outfile"

(BTW what I've seen should run fine with sh, ie #!/bin/sh or #!/bin/bb if you haven't linked it;)

Note the quoting of "$url" which is important for maintainability. Using a variable for output files is always to be preferred ime, as it can easily be edited at the top of the file without having to dig around in the script. OK, not so important for a small scriptlet, but small scripts tend to get expanded, or used in novel ways. If they are robust then they can be (and thus fit into the Unix ecosystem.) If not, they just become a source of frustration, similar to C code that doesn't check error-returns. If you quote consistently, you can cut and paste bits of script around, and as in the grep above, your intentions are clear on sight.

The point is to get into the habit of writing robustly from the beginning: #bash is excellent for that, since you see lots of one-liners flying aorund, and instantly getting corrected (or solved more efficiently.) That means you quickly learn of the pitfalls and better, see them happening along with how to fix them, and explanations (usually with a specific url that explains the issue.)
They are also very good on portability across operating-systems, let alone Linux distros.

Using ${foo} when there is nothing following it which would get confused with the variable name, ie an alphanumeric or _, eg: echo "${foo}bar", is considered pretentious or ignorant by old-school scripters (sometimes both). Too often, it masks a confusion whereby beginners think they don't need to quote with ${foo}; or why would you prefer it to $foo? Arguing for two extra shifted keystrokes in the name of "consistency" with a rare case is just academic wankery to a working scripter. Any decent editor can syntax-highlight the variable, even nano. Hell there are websites that syntax highlight them in pastebins, so the "it looks nicer" argument, is a non-starter as well, irrespective of what ebuilds look like. (That is the only justification I've seen and it's laughably amateur.)

That in fact is the one area I could never argue with greycat on: and believe me, he used to say some terrible things about Gentoo developers and users ;-)

Regards,
steveL.
_________________
creaker wrote:
systemd. It is a really ass pain

update - "a most excellent portage wrapper"

#friendly-coders -- We're still here for you™ ;)
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6227
Location: Room 101

PostPosted: Mon Mar 18, 2013 2:26 pm    Post subject: Reply with quote

steveL wrote:
So instead you reinforce a bad habit, just when they're starting out?

steveL ... I'm not going to argue, after the fact, that its correct, it plainly isn't, its uuoc, and I admited as much, at the time I didn't think much of it, which is perhaps an expression of my own "bad habits", but honestly it was mostly a case of not wanting to loose the reader with digressions. That may be lazy, and given the times I've pointed out uuoc elsewhere it certainly is, but really I sensed the user wasn't clear about what the script did, and rather than provide a tutorial I'd try and make it as transparent as possible. In retrospect I'd probably have been best to use 'read <' and just leave it unexplained, and frankly I wish I had, but it wasn't a case of "reinforcing", it was expediency more than anything.

steveL wrote:
khayyam wrote:
I tried to keep the script simple [...]

Yeah and that's just unnecessary: all you end up doing is giving the impression that that's what shellscripters would do, and it blatantly is not. Better just to simply explain what you're doing and why you're doing it, if it isn't obvious.

How many threads on the forum do you think there are in which the OP has long since gone silent due to incomprehension, and which the finer points of fact are discussed, disected, explained in detail, etc ... I'd hazard quite a few. I'm generally all for such discussion, but I try to keep in mind the OP. Thats not a personal criticism, most anyone reading this thread might gain something from the examples you posted, but there are cases in which its not self explanatory, and the examples, and language used, trigger incomprehension. The context is important, and although jonfr may have some interest in learning shell scripting, and so benifit from best practices, from their response "so this dumps the data into a file locally" (when this wasn't what was happening at all) I took as a sign that I should keep things as basic as possible.

steveL wrote:
khayyam wrote:
Code:
data="$(wget --quiet -O - ${url} | grep $(date +%Y%m%d) | wc -l)"
echo "${data}" >> /path/to/file.log

could simply be:
Code:
wget --quiet -O - "$url" | grep "$(date +%Y%m%d)" | wc -l >> "$outfile"

Indeed, that'd be better ...

steveL wrote:
Using ${foo} when there is nothing following it which would get confused with the variable name, ie an alphanumeric or _, eg: echo "${foo}bar", is considered pretentious or ignorant by old-school scripters (sometimes both). Too often, it masks a confusion whereby beginners think they don't need to quote with ${foo}; or why would you prefer it to $foo? Arguing for two extra shifted keystrokes in the name of "consistency" with a rare case is just academic wankery to a working scripter. Any decent editor can syntax-highlight the variable, even nano. Hell there are websites that syntax highlight them in pastebins, so the "it looks nicer" argument, is a non-starter as well, irrespective of what ebuilds look like. (That is the only justification I've seen and it's laughably amateur.)

You have me ... I guess I do it out of habit. None the less its a $var, and the curley braces neither add nor distract from that fact. Your also right wrt quoting, though given the asignment of var is quoted they should be safe.

best ... khay
Back to top
View user's profile Send private message
jonfr
Veteran
Veteran


Joined: 20 Jul 2003
Posts: 1008
Location: Denmark

PostPosted: Fri Mar 22, 2013 6:34 pm    Post subject: Reply with quote

What is the best way for me to get information from this web page here into mrtg plot? I have been trying to do so with the script that is in the first post. But it has not been working. Since it doesn't seems to count the earthquakes. The data is on page 3 on this web page when using text browser.

Web site in question, http://www.02.ign.es/ign/layoutIn/volcaListadoTerremotos.do?zona=2&cantidad_dias=10
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Gentoo Chat All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum