Davis Vantage Pro weather station: software for Python and Linux

This page will describe the software I'm writing to pull information from the Vantage Pro to Linux and then put it on the Web. Currently it - and the software - is under construction. The code isn't in CVS or even in a tar ball on the Web, but I'm can tar up the latest on request.

Linux, Python, and MySQL

These three products are great, flexible, and easy to work with. (Well, those of us who think Unix is user-friendly admit that Unix is picky in choosing its friends.) I use Linux because I've used Unix since 1980 or so. I use MySQL because I had to convert the NH Mensa mail list from a flaky, mostly Asthon-Tate compatible dbase II system to something that had a future. Unfortunately, MySQL and similar "modern" databases don't come with a semi-decent scripting language, so I had to rewrite a big slew of scripts. That brought me to Python, a language that works well on Windows too. If you're considering learning Perl or Java, learn Python first and you may discover you don't need the others.

One of my reasons to buy a weather station was to have an excuse to write more Python code. The Davis Vantage Pro has a fairly simple protocol and is publically documented. All in all, the Vantage Pro is a good system to access from a home computer. The things I knew I needed to do were:

In addition to the end goals above, there's a fair amount of infrastructure that includes its own goals and small projects:

Database and tables

I'm only using a single MySQL database (wx) with these tables:

Program Flows

I want to come up with a matrix that shows the Python programs used, the MySQL tables affected, and other files affected. Collecting that will take a bit of time, so I'll just write some of the supporting text for now.

curupdate.py - read recent data from console and save in registry.

This reads the high/low data and saves the day's high and low temperature date in wx.registry. It also reads the current data via the LOOP request and saves the interesting data in the registry. This is used by several other programs, notably webupdate.py.

daily_update.py - bring wx.daily up-to-date

This reads recent data in wx.raw and creates a summary for each day in wx.daily.

rawupdate.py - copies Vantage Pro data to wx.raw

I have my Vantage Pro console logging data every 10 minutes. Cron runs a make command every 20 minutes to update the WWW site, and this is the first step in the process. Rawupdate.py gets the latest data from the console and adds it to the wx.raw table. It remembers the date and time of the last sample in a text file, soon to become an entry or two in wx.registry. The basic flow is to connect to the console, recover the timestamp, read new data from the console (via the "dmpaft" command), check each sample for reasonableness and add them to wx.raw. Samples are generally rejected if the wireless link between the sensor suite and console dropped a lot of data. Currently I'm just a few feet from the edge of the range and rain or snow plays havoc with reception. It's easier to just drop the sample than to keep the good data (e.g. barometric pressure) and handle the missing data.

webupdate.py - create gnuplot and sed scripts

This fetches the last 24 hours worth of data from wx.raw and creates a gnuplot script and a data file to create graphs for the Web site. It gets current data from the console via a "loop 1" command and creates a sed script to convert a template file into a HTML page text for the current conditions.

pngfix.c - improve .PNG file characteristics.

Obviously the next step is to run gnuplot and sed. However, there are a number things that gnuplot does that I don't particularly like. The .PNG files it creates have a bright white background. That should be saved for highlights in photographs! Also, gnuplot uses light colors for its plotted lines. Yellow on white doesn't work. Finally, lack of control over line colors prevents me from using colors in the order I want.

Instead of doing the sensible thing and fixing gnuplot, I wrote a C program that patches the .PNG files created by gnuplot. The Makefile calls pngfix to dress up all the plots then runs ftp to get the new data on the server.

whdhupdate.py - update WHDH weather spotters' data

Cron runs this four times a day before newscasts to update their spotter data page. This gets the recent temperature and wind speed from registry values written by curupdate.py.

whdh.py - print Email template for WHDH weather Email list

This gets current weather data and yesterday's high and low, then prints a template for an Email post to WHDHWX. I usually only post when I have something to say beyond what the raw data is.

spotupdate.py - update weatherspotters.com

This updates the weatherspotters.com site with the current temperature and high/low data for the day. It also wants snow/ice/rain, but I'm not collecting that yet.

query.html, cgi-bin/wx_fetch

Query.html displays a couple web forms for querying the raw and daily tables. The forms let you select a starting date, a duration, and the variables to display. Clicking Submit sends the data to wx_fetch which extracts the data from the table and has gnuplot create a .png file. wx_fetch then returns HTML code to access the .png file.

mysqlreg.py, reg.py, regtest.py

mysqlreg.py implements a key/value registry that is used by most of the main code. The registry exists as a mysql table named "registry". reg.py is a command-line program that lets you add, delete, view, and change entries. regtest.py is a limited unit test program.

vantage_pro.py - Davis protocol module

This code understands the Vantage Pro protocol. A class for each protocol request holds the data from the replies. This is imported into any program that talks to the Vantage Pro console.

wxmath.py - library of weather related math routines

Routines include:

Occasional use programs and quick&dirty hacks

daily.py - Create High/Low plots

This reads some data from wx.daily and creates high/low summary for a range of months in www/highlow.png.

depthupdate.py - Probably useless

I wrote this to copy snow depth from a .csv file emitted by gnumeric to wx.daily. It was useful for moving a season's worth of data, it's not so convenient to use for a few days' data.

export.py - convert WeatherLink date/timestamps to Mysql

Davis's WeatherLink program exports uses an American "mm/dd/yy hh:mm{a,p}" style timestamp, whereas MySql use the odometer-style "yyyy-mm-dd hh:mm:ss". This program export.txt (created by WeatherLink) and writes export.mysql. I forget how I got that into MySQL.

waketest.py - early protocol test program

This is unnecessary for normal operation. It sends various protocol requests to the console and reports progress.

Contact Ric Werme or return to his home page.

Last updated 2005 Nov 25.