Davis Vantage Pro weather station: software for Python and Linux
This page will describe the software I'm writing to pull information from
the Vantage Pro to Linux and then put it on the Web. Currently it - and the
software - is under construction. The code isn't in CVS or even in a tar
ball on the Web, but I'm can tar up the latest on request.
Linux, Python, and MySQL
These three products are great, flexible, and easy to work with. (Well,
those of us who think Unix is user-friendly admit that Unix is picky in
choosing its friends.) I use Linux because I've used Unix since 1980 or so.
I use MySQL because I had to convert the NH Mensa mail list from a flaky,
mostly Asthon-Tate compatible dbase II system to something that had a
future. Unfortunately, MySQL and similar "modern" databases don't come with
a semi-decent scripting language, so I had to rewrite a big slew of scripts.
That brought me to Python, a language that works well on Windows too. If
you're considering learning Perl or Java, learn Python first and you may
discover you don't need the others.
One of my reasons to buy a weather station was to have an excuse to write
more Python code. The Davis Vantage Pro has a fairly simple protocol and is
publically documented. All in all, the Vantage Pro is a good system to
access from a home computer. The things I knew I needed to do were:
Provide WWW pages with current and historical weather data.
Davis's WeatherLink software and others do this, I didn't find a good
starting point for Linux software. I didn't look very hard, as I wanted to
write code anyway. Gnuplot makes it easy to come up with graphs to put on
the WWW.
Maintain long term computerized records.
I still haven't decided if I want to join a NOAA affiliated program.
Automatically update other archives.
Todd Gross, a TV met at Boston's WHDH has worked long and hard to develop a
spotter network, a process that is ongoing. WHDH now accepts weather data
by phone, a private Yahoo group's Email list (WHDHWX), and a WWW form that
updates an interactive
area data page. I have code that runs via cron to submit the post to
update the WHDH page. Another effort at http://newxspotters.net/ is used by
WHDH and is available to their weather spotters and others beyond the
area WHDH covers. My software is behind the data logging parts of that
site but is not described on the web yet.
Join the Weather Underground.
I thought I'd be more involved with this by now, but the above wound up
getting priority.
In addition to the end goals above, there's a fair amount of
infrastructure that includes its own goals and small projects:
Fixing gnuplot's graphs.
I mentioned above that it's easy to make graphs for the Web with gnuplot.
However, coming up with graphs that look good requires a bit of hackery. I
first ran into that with .GIF files from Gnuplot and wrote a C program that
changes colors and transparency. I bashed that into a new verion that does
the same thing with .PNG files. It's still in C.
A MySQL registry-like table.
Anyone who started working with Unix in PDP-11 days appreciates the ASCII
configuration files it has. Being able to work with them with the rich tool
set that comes with Unix opened the door to all sorts of administrative
scripting tools. That still holds true today. However, I discovered that
in creating the "kludge tower" of programs to do everything I needed, I was
creating a number of small files. For example, the WHDH site needs current
temperature and wind speed. The program that updates my WWW site saved
those in two small files. A cron entry picked them up and passed them to
another program that sent the form to WHDH. Trying to keep several pieces
of changing data in a file became a tad messy. Since everything is working
with a database, I wrote a Python module to maintain a key/value registry
and updating each takes just a line of code. Of course, there are functions
to export and import files with colon separated fields so you can still work
in classic Unix style.
Distractions (mostly welcome).
I've learned a bit more about weather than I expected while working on this
code. The learning and writing code based on that took some time I wasn't
planning to do right away. However, it's time well spent. One nice thing
about a sizable personal project like this is I'm not constrained by
customer-centric schedules. Well, almost. The first pass of the WWW update
code didn't use MySQL at all because I wanted data on the web ASAP without
having to boot Windows and run WeatherLink there.
The Vantage Pro protocol doesn't report all derived data. For example,
dew point is derived from temperature and relative humidity. In hunting
down the formulae and theory behind that, I came across formulae to compute
the frost point, something that makes more sense in subfreezing weather here
at ground level with condensation nuclei everywhere I look. So I compute
that too.
Todd Gross has a home snow making system and lamented that you need wet
bulb temperatures around 29°F or below to make snow, but that few
(none?) weather stations report wet bulb. Computing it turns out to be an
iterative process, so I wrote a program to make a colorized table to convert
air temperature and dewpoint to wet bulb.
The new
windchill formula is much, much more sensible than the old one. The old
one wasn't worth computing, the new one is. I'm not fond of a couple of the
policies behind the new one, namely that wind chill is a
dimensionless integer and is undefined at wind speeds 3 mph and less.
However, they got a couple things right, especially trying to get the wind
chill math to make wind chill nearly equal air temperature at walking speed.
Database and tables
I'm only using a single MySQL database (wx) with these tables:
raw
This holds most of the data the weather station logs each 10 minutes.
I'll refer to it as "wx.raw".
daily
This has one entry per day for things like high and low temperatures,
snow depth, etc.
registry
This holds data that is passed between programs. Each record has
fields for the key, values, and type (string, integer, or real).
Program Flows
I want to come up with a matrix that shows the Python programs used, the
MySQL tables affected, and other files affected. Collecting that will take
a bit of time, so I'll just write some of the supporting text for now.
curupdate.py - read recent data from console and save in registry.
This reads the high/low data and saves the day's high and low temperature
date in wx.registry. It also reads the current data via the LOOP request
and saves the interesting data in the registry. This is used by several
other programs, notably webupdate.py.
daily_update.py - bring wx.daily up-to-date
This reads recent data in wx.raw and creates a summary
for each day in wx.daily.
rawupdate.py - copies Vantage Pro data to wx.raw
I have my Vantage Pro console logging data every 10 minutes. Cron runs a
make command every 20 minutes to update the WWW site, and this is the first
step in the process. Rawupdate.py gets the latest data from the console and
adds it to the wx.raw table. It remembers the date and time of the last
sample in a text file, soon to become an entry or two in wx.registry. The
basic flow is to connect to the console, recover the timestamp, read new
data from the console (via the "dmpaft" command), check each sample for
reasonableness and add them to wx.raw. Samples are generally rejected if
the wireless link between the sensor suite and console dropped a lot of
data. Currently I'm just a few feet from the edge of the range and rain or
snow plays havoc with reception. It's easier to just drop the sample than
to keep the good data (e.g. barometric pressure) and handle the missing
data.
webupdate.py - create gnuplot and sed scripts
This fetches the last 24 hours worth of data from wx.raw and creates a
gnuplot script and a data file to create graphs for the Web site. It gets
current data from the console via a "loop 1" command and creates a sed
script to convert a template file into a HTML page text for the current
conditions.
pngfix.c - improve .PNG file characteristics.
Obviously the next step is to run gnuplot and sed. However, there are a
number things that gnuplot does that I don't particularly like. The .PNG
files it creates have a bright white background. That should be saved for
highlights in photographs! Also, gnuplot uses light colors for its plotted
lines. Yellow on white doesn't work. Finally, lack of control over line
colors prevents me from using colors in the order I want.
Instead of doing the sensible thing and fixing gnuplot, I wrote a C
program that patches the .PNG files created by gnuplot. The Makefile calls
pngfix to dress up all the plots then runs ftp to get the new data on the
server.
whdhupdate.py - update WHDH weather spotters' data
Cron runs this four times a day before newscasts to update their spotter
data page. This gets the recent temperature and wind speed from
registry values written by curupdate.py.
whdh.py - print Email template for WHDH weather Email list
This gets current weather data and yesterday's high and low, then prints a
template for an Email post to WHDHWX. I usually only post when I have
something to say beyond what the raw data is.
spotupdate.py - update weatherspotters.com
This updates the weatherspotters.com site with the current temperature and
high/low data for the day. It also wants snow/ice/rain, but I'm not
collecting that yet.
query.html, cgi-bin/wx_fetch
Query.html displays a couple web forms for querying the raw and daily
tables. The forms let you select a starting date, a duration, and
the variables to display. Clicking Submit sends the data to wx_fetch
which extracts the data from the table and has gnuplot create a .png
file. wx_fetch then returns HTML code to access the .png file.
mysqlreg.py, reg.py, regtest.py
mysqlreg.py implements a
key/value registry that is used by most of the main code.
The registry exists as a mysql table named
"registry". reg.py is a command-line program that lets you add,
delete, view, and change entries. regtest.py is a limited unit test
program.
vantage_pro.py - Davis protocol module
This code understands the Vantage Pro protocol. A class for each
protocol request holds the data from the replies. This is imported
into any program that talks to the Vantage Pro console.
wxmath.py - library of weather related math routines
Routines include:
Convert degrees to compass point. (e.g. 45 to NE).
Convert wind velocity to and temperature to wind chill.
Convert temperature and humidity to dew point and frost point.
Occasional use programs and quick&dirty hacks
daily.py - Create High/Low plots
This reads some data from wx.daily and creates high/low summary
for a range of months in www/highlow.png.
depthupdate.py - Probably useless
I wrote this to copy snow depth from a .csv file emitted by gnumeric to
wx.daily. It was useful for moving a season's worth of data, it's not
so convenient to use for a few days' data.
export.py - convert WeatherLink date/timestamps to Mysql
Davis's WeatherLink program exports uses an American "mm/dd/yy hh:mm{a,p}"
style timestamp, whereas MySql use the odometer-style "yyyy-mm-dd hh:mm:ss".
This program export.txt (created by WeatherLink) and writes export.mysql.
I forget how I got that into MySQL.
waketest.py - early protocol test program
This is unnecessary for normal operation. It sends various protocol
requests to the console and reports progress.
