AWRA banner
Advancing Water Resources Research and Management

Symposium on Water Resources and the World Wide Web
Seattle, Washington, December 5-9, 1999

Interactive Graphical Access to Real time and Retrospective Precipitation Data

William R. Moninger
Edward I. Tollerud
Mark W. Govett

NOAA Forecast Systems Laboratory
Boulder, Colorado

Index

Abstract and Introduction

small image of the realtime precipitation display
Click on the image for a larger view (29 Kb)
NOAA's Forecast Systems Laboratory has developed two Java based interactive displays of precipitation data, which may be found at http://precip.fsl.noaa.gov/ [external link]. These provide access to hourly and daily precipitation data from nearly 10,000 sites in the continental United States. One display, which we will call the "real time display," shows hourly data from the Hydrometeorological Automated Data System (HADS), as well as data from stations that report once per day. The other, which we will call the "HPD display" provides access to hourly data for periods of up to 50 years, available as the "Hourly Precipitation Dataset" (HPD) on CD-ROM from the National Climatic Data Center.

Both displays show stations on a national map, are color-coded to indicate precipitation amount and shape-coded to indicate how often the station reports and whether its data is complete or incomplete for the time period. For any station under the cursor, detailed data are displayed. Zooming is possible, and an overview is provided to help locate and adjust the zoomed in region. River and county map backgrounds may also be displayed, and the display may be restricted to only show those observations in user selected precipitation amount ranges.

When a station is clicked on, a time-series of data is shown if available, and by moving the cursor across the time-series, individual data values are revealed. Dragging across the time series generates precipitation subtotals.

 We briefly discuss the data displayed, and more extensively discuss our design criteria and the resulting software, which we developed as Java applets.  We believe that our experiences and some of the techniques we have developed for delivering and displaying these data may be useful to others who wish to display geographically based time series data.

Motivation

Under FSL's mission of technology transfer, we have investigated and developed advanced techniques for processing and displaying atmospheric and hydrologic data. As the usefulness of the World Wide Web became apparent, we decided to explore the Web as a vehicle for the distribution and display of these data. We chose to work with precipitation data for several reasons.

The Data

Real time Hourly Data

HADS [external link] provides data from over 5000 hourly rain gauges over the Continental United States. These data are automatically telemetered to the Office of Hydrology, from which the individual River Forecast Centers (RFCs) acquire them (they consider themselves users rather than producers of the HADS). The individual gauges are operated by various agencies, including the Bureau of Land Management, the United States Geological Survey, and the National Weather Service. Although efforts are now being coordinated at the Office of Hydrology to provide some overall quality control of HADS, the data stream we currently receive does not include any QC, to our knowledge. We have so far chosen to display HADS precipitation amounts as is, without any additional QC. In this way, the web site is useful for monitoring the quality of precipitation reports, particularly the overall frequency and severity of problem data sites. More information.

Real time 24-h Data

The 12 RFCs in the lower 48 states are the originators of the daily data displayed at our web site. Included in the 24-h totals are automatic (ASOS [external link] - Automated Surface Observing System) observations at Weather Forecast Offices, manual observations by cooperative observers overseen by weather service personnel, totals generated elsewhere from hourly gauges that are part of HADS, and totals from gauges at mesonets or other sites that report to the RFCs.

 These observations are quality controlled at individual RFCs before being transmitted to NCEP and from there to us. For both automatic and manual readings, the QC procedures include the use of radar observations to see if the hourly reports could be confirmed by the presence of radar returns as well as tests to remove obviously excessive values. Individual RFCs have their own quality control procedures, geared to the requirements of their own region, hence different RFCs place different emphasis on quality control. More information

HPD Data

The observations on the 50-year Hourly Precipitation Data CD-ROM were taken by observers at primary stations, secondary stations, and cooperative observer stations operated by the NWS and the Federal Aviation Administration (FAA) between August 1948 and the present. Approximately 5,500 stations have recorded precipitation data at some time during the period of record.

 The data are a combination of original observations of hourly and daily accumulated precipitation. Precipitation values are checked and edited as necessary by an automated and manual edit. More information, (Hammer and Steurer, 1997).

A program in the C language was initially developed to provide access to data on the CD-ROM. This program was written in C for speed, and portability.  This program remains the sole method for retrieving data from the CD-ROM.  It has several data output formats, one of which is a format that can be read by the Java display software. Thus, retrieving and displaying data from the CD-ROM is a two-step process: first, run the C program to create files for display, then run the Java display applet (locally--no Web access is required, only a web browser).   This program is available only for Unix platforms. More information [external link].

Design Considerations for the Display

To display these data, we desired to develop systems that

Display Software

Based on the considerations above, we chose to develop Java applets, using version 1.0.2 of the Java language. (Java version 1.1 existed when we started, but could not be run in generally available web browsers: Netscape 3 and Internet Explorer 3.)

 The two applets are very similar. The HPD display reads local files generated by the CD-ROM access software on the user's local computer, while the real time display reads files generated once per day on one of FSL's Web servers. In both cases, manipulations of the displayed data are done locally on the user's computer.

 On line versions of these applets are available at http://precip.fsl.noaa.gov/ [external link]. For those without access to the web, here is a local version of the HPD display.

Mapping and Backgrounds

Data are displayed on a Lambert conformal conic projection, tangent to the globe at 20 degrees north latitude, and centered at a longitude of 95 degrees west. We call this coordinate system 'java-conus-1'.  The background map consists of state and county boundaries as well as major rivers.

 The map is generated in an off screen buffer with size 600 pixels wide by 350 pixels high. When new data or backgrounds are displayed or the map is zoomed or panned, the off screen buffer is regenerated; when the applet must be repainted (which occurs every time the mouse moves over the map, or when a portion of the map is revealed after having been hidden by another window), the off screen buffer is copied to the screen. Then, additional information, if required, is plotted directly to the screen.

Zooming and Panning

Zooming the map is accomplished by dragging the mouse diagonally across the region of interest. While the mouse is being dragged a red rectangle appears indicating the extent of the region to be zoomed. Because the aspect ratio of the window showing the map is fixed, the aspect ratio of the red rectangle is also fixed, so that as the cursor is moved, it isn't always on a corner of the rectangle. Initially, this surprises everyone, and takes a bit of getting used to. Nonetheless, we haven't been able to devise a more intuitive zooming strategy, and users seem able to quickly understand how to zoom.
 
 
zoom box shown on the precipiation display zoomed-in display
Click on either image for a larger view

An 'unzoom' button is provided to undo the previous zoom. Initially, only a 'reset' button was provided, which reestablished the original national map. However, we discovered that users often zoomed several times in order to disambiguate nearby stations, and they desired a way to incrementally unzoom. So, we created the 'unzoom' button which undoes one zoom at a time. When we implemented this, we also found we needed to again provide the 'reset' button, because users often wanted to be able to reestablish the original map without having to undo a long sequence of zooms.
 
 
overview window
Overview - actual size
An 'overview' button creates a small window showing the national map, with the current zoomed region highlighted as a red rectangle. The user can use the cursor to drag this rectangle to another location, thereby 'roaming' the region shown in the main window. This is particularly useful for making minor adjustments to the map. Initially, we also allowed the user to change the zoomed region by dragging an edge of the small rectangle in the overview window, but we later eliminated this feature because we thought it unnecessarily complex and confusing.

Color/Shape coding and selective display

precip map showing a limited set of precipitation values
Click on the image for a larger view
Plotted data are color coded by precipitation amount, with the precipitation value corresponding to each color indicated by an interactive color bar below the map. If users desire to display only stations reporting a particular range of precipitation, they can click on the corresponding color on color bar to do so. Dragging the cursor across the color bar permits the display of a wider range of precipitation values.  Colors that will not be displayed are x'ed out on the color bar, as is shown on the figure at right.  Although we have found this to be useful functionality, the user interface is not yet intuitive enough for our liking. It is not obvious that this functionality exists, nor how to restore display of all the data (clicking again on a displayed color on the color bar). We invite suggestions about how to improve this.

 Data are plotted as solid or hollow symbols to indicate whether the time series is complete or incomplete, respectively. In addition, for the real time display, data are shape coded. Squares indicate hourly data, diamonds indicate 24-h total data, and circles indicate stations that appear in both data streams.

Readout of Data at the Cursor Location

data readout near the cursor Initially, text fields above the map indicated data and metadata for any station underneath the cursor. The data shown were latitude, longitude, station name, amount of precipitation, and geographic description. We eventually discovered, however, that it was more natural for most of this information to appear near the cursor location rather than above the map, because the user's eye is generally following the cursor.

Currently only the cursor latitude and longitude are shown above the map; the other data are shown at a fixed location with respect to the station under the cursor.

 This functionality is implemented in the following way. A 2 dimensional array, called the 'data_pointer array' with the same dimensions as the number of pixels in the map is filled with either pointers to the station corresponding to a particular (x,y) location (for a given zoom and roam), or with a 'no station' flag. Each time the cursor is moved, nearby locations in the data_pointer array are interrogated. If any of these locations contains a pointer to a station, that station's data are accessed and plotted.

Initially, the strategy was that only the location in the data_pointer array under the cursor was interrogated, but this required the cursor to be directly over a station (to the nearest pixel) for the data to be plotted, and this was too sensitive to the mouse position for convenient use. Our second strategy was to fill multiple adjacent positions in the data_pointer array with pointers to each station. For instance, each station would 'occupy' a region in the data_pointer array 5 pixels on a side. This made it easier for users to place the cursor (approximately) "over" a station, but caused stations to overlap in regions of high station density.

 The current strategy is to have only one entry in the data_pointer array for each plotted station, and when the mouse moves to a new location, we interrogate locations in the data_pointer array in an outward moving spiral, starting with the location directly under the cursor and moving outward until either a station is found, or a preset limit is reached. (Currently, that limit is set as 4 pixels on either side of the cursor position.) This strategy minimizes station overlap, allows the mouse to be near rather than directly over a station, and always chooses the station closest to the cursor. Here is the code that accomplishes this.

 This strategy, although complex, runs apparently instantly even on relatively slow computers (such as a 166 MHz machine).

Time Series Plots

For any station that provides a time-series (everything but the once per day stations shown by the real time display), clicking the mouse while the cursor is over the station will produce a time series window. The abscissa of the time-series plots is an approximately logarithmic scale running from 0.01 inches to 15 inches. (The relationship is

y ~ (1+p^.33)*log(p)

where y is abscissa value and p is precipitation value.) The ordinate is the time, with increments which vary from hours to days and months. (The latter two are only available on the HPD display.)

 Data for each subinterval of the plot are reported as histogram bars, or as one of several different types of missing indicators. (Data can be missing because they were not reported, or were reported as missing, incomplete, or accumulating.)

time series histogram showing data readout at cursor Because it can be somewhat difficult to read the exact precipitation amount or time represented by a particular histogram bar, the amount and time are displayed as text near the cursor whenever the cursor is moved across the time-series. The figure shows an example in which the information for hour 21 is highlighted..

 If the user desires to know the precipitation during a subinterval, this can be accomplished by dragging the cursor across the desired subinterval. When the mouse button is released, a new window will appear indicating the total precipitation for the subinterval, and providing information about any data gaps.

In order to effectively deal with this time series information, and to accommodate the wide variety of time intervals that can be displayed from the HPD CD ROM, we found it necessary to develop our own Java class for dealing with dates and times.  Unfortunately, the date classes provided by both Java 1.0.2 and Java 1.1 were either insufficient or too tightly linked to local time (with its complexities having to do with daylight savings time) for our needs.

Data Structure

Data used by the display applets are of several types Most of these data are sent to the applets in compressed format, to minimize download time. (Download time is not critical for the HPD applet, because it runs locally and loads data directly from the local disk, but we wanted to keep the applets and formats as similar to one another as possible.)

Geopolitical Boundaries

The map background comprises 325 state boundary segments, 537 river segments (at 15-km resolution), and 3,860 county boundary segments (at 20-km resolution).  These resolutions were established by eye as the minimum acceptable resolution for our uses.  In addition, these data were premapped to the java-conus-1 coordinate system, so that the data could be stored as short integers (2 bytes) rather than floating point data (4 bytes). The data were additionally compressed by using a format that only carries deviations from the previous point, rather than the full java-conus-1 location for each point. Here are the details of this format.

By reducing the resolution and carefully choosing a data format, we were able to decrease the size of the files holding the background maps to 12% of their original size. Since computational time to process this map background data is negligible compared to download time for the data over modem lines (even at 56.6 Kb), this reduced the time startup time for the applet considerably.

In addition, to speed up initial download, county and river boundaries are downloaded in the background (in a separate, lower priority thread) after the applet itself is downloaded. (Unfortunately, due to weakness in implementations of Java's multithreading functionality on several platforms, the applet operates noticeably slower until the county and river data are finished downloading.)

Station Metadata

For the real time display applet, station metadata are downloaded as a separate file. For the HPD applet, station metadata are downloaded with each time interval selected. The reason for this difference is that the 50 year HPD data has much more metadata (many stations moved over the years) than does the real time data, and most of the HPD metadata are irrelevant to any particular time interval.

Although we carefully developed a metadata format that had no extraneous information, we discovered that we could further reduce the size of the metadata file 45% by compressing it with the gzip file compression scheme. Unfortunately, this conflicted with our requirement to use Java 1.0.2 rather than Java 1.1, because the former does not support gzip decompression. We could have developed our own Java 1.0.2 decompression class, but the ever decreasing number of users unable to use Java 1.1 did not seem to justify this. Nonetheless, we didn't want to render the applet inoperable by those users restricted to Java 1.0.2.

 We therefore chose to take advantage of Java's ability to conditional load classes.  We now test the user's version of Java, and, if the user's version of Java is 1.1 or greater, we load a class at run-time that uses Java 1.1's "GZIPInputStream" class to load and decompress the gzipped version of the station metadata file. If the user's version of Java is less than 1.1, the uncompressed metadata file is loaded instead. Here is the code that accomplishes this.

Precipitation data files

For the real time display, daily precipitation totals are stored in separate files from hourly time-series data; for the HPD display, for which download time is not an issue, time interval totals are stored along with time-series data in ascii files created by the CD ROM access software. These formats are generally straightforward.  The structure of the hourly time-series format may be noteworthy, however. The hourly time-series data are only needed for two cases: when there is nonzero rain during the 24-h period, or when there is no rain, but there are missing hours (the time series is needed to record which hours are missing). The hourly time series format is designed to store only the minimal amount of data required to reconstruct the time series in each of these cases.

Strengths and Limitations of Java Applets

Strengths Weaknesses

Summary

We have reported on two Java applets that display real time and retrospective precipitation data. Java has been a good choice for the display technology, allowing relatively sophisticated functionality with minimal up front cost for the user in terms of downloading and installing software. We have used Java version 1.0.2, which has allowed wide accessibility to the displays, with limitations that we have found to be relatively minor.

 Algorithms we have developed that we believe may be widely useful include our strategy for displaying data values near the cursor, conditional loading of Java 1.1 classes in support of decompression, and a class to manipulate UTC dates.

Acknowledgments

We wish to acknowledge the following

References

Govett, Mark W., Edward I. Tollerud, and William R. Moninger (1997): A Demonstration of the Access and Display of Fifty Years of Hourly Precipitation Data Available on CD-ROM [external link] 10th Conference on  Applied Climatology (Reno, NV, 20-24 October)

Hammer, G.R. and Steurer, P.M., 1997: Data set documentation for Hourly Precipitation Data. NOAA/NCDC TD3240 [external link] Documentation Series, Asheville, NC, 18 pp.

Tollerud, E. I. and W. Moninger (1996): On The Consistency Of National-Scale Analyses Using Existing Precipitation Gage Networks. Proceedings, Second International Scientific Conference on the Global Energy and Water Cycle

Tollerud, Edward I., Mark W. Govett, William R. Moninger, Peter M. Steurer (1997): New Access and Display Routines for Hourly Precipitation Data and Metadata using CD-ROMS and the World Wide Web. [external link] 10th Conference on Applied Climatology (Reno, NV, 20-24 October)

Tollerud, E. I., (1999) Combining precipitation networks: The effect of instrumentation and site distribution differences. [external link] 11th Conference on Applied Climatology.
 

End note

Some limited data in the "50 year" database actually extend back as far as 1900.  But the data is commonly called the "50 year database," so we continue that usage here.

AWRA
Symposium TOC AWRA Home page

Maintainer: AWRA Webserver Team
Copyright © 1999 American Water Resources Association
Content produced by U.S. Government employees as part of official duties is not subject to copyright.