How would you build a weather station reporting website?

DingbatCA · Guru Joined: 07 Jul 2004 Posts: 318 Location: Portland Or

How would you build a weather station reporting website?
I have about 50 weather stations. Each one calls home with a cell phone, and reports its current values. The data is not big... only a few 18K a minute... But 18K*1440 is 25MB of data a day, each. It get to be 1.3GB a day for the raw data for all 50 stations. I am using lzma compression to solve that issue.

Logical order of operations:
Weather station make a call (Think of it like a wget)
cgi (shell script) script on the server side, reads in the data, and places it in /home/weather/station001/07012009/1703
Station name/date code/time code(with values inside)
cgi script returns the current date/time and if the data was received successfully, for the weather station to sync time, or resend the data.

I then pull all the time codes into excel to make graphs. This is slow and a pain.

I want to build a full web interface. I am paid by the hour :-), so I don't mind learning a new language, like ruby, or ajax. I am guessing a MySQL database to store the data will be a good approach? I would like the web interface to be dynamic; don't want to hit refresh every 5 min.

The server: I have Gentoo loaded on a Athlon X2 2.4GHz 64-bit based system. 4GB ram, an 1TB of mirrored disk.

How would you do it? What language would work best? Would you use a data base, if so , which one, and why? Size problems, its going to get big, how do I work around that? Not looking for code, just trying to build my plan of attack, before I spend a month working on building this web site.
-DingbatCA

poly_poly-man · Posted: Wed Jul 01, 2009 6:54 pm Post subject:

mysql database is perfect for the data.

ajax/javascript is necessary for some of the interfacing (for it to be dynamic)... python and php are good languages for the underlying layer.
_________________
iVBORw0KGgoAAAANSUhEUgAAA

avatar: new version of logo - see topic 838248. Potentially still a WiP.

jlpoole

NeddySeagoon · Posted: Sun Apr 22, 2012 9:27 pm Post subject:

jlpoole,

If you want to limit the size of the databse rrdtool is an option. It keeps the last n samples, where n is chosen by you and makes it easy to draw graphs with various timebases.
Its not what you want if you are going to allow the database to grow without limit.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

SadLittlePony · n00b Joined: 22 Apr 2012 Posts: 10

My father runs a weather station. His website is here. http://www.frontiernet.net/~mrobo/ you should hit him up with an email.

You would make him feel important

jlpoole · Posted: Mon Apr 23, 2012 12:20 am Post subject:

SadLittlePony · n00b Joined: 22 Apr 2012 Posts: 10

the website should have been a weather page kinda bland looking but his email is http://www.frontiernet.net/~mrobo/mrobo_at_gmail.png in this picture thingy. he does it that way for security. and though the page looks boring there is tons of information. and yes he built the equipment himself