BSRN Toolbox

The Baseline Surface Radiation Projects (BSRN) makes use of the so called station-to-archive format to share the data of one station for one month with the BSRN community. This format cannot be understood without reading the BSRN format manual. So the BSRN Toolbox was developed to convert this format to more readable dataset tables and for creating PANGAEA import files. These tables can be opened with a spreadsheet program like Microsoft Excel or OpenOffice and with PanPlot to visualize the data very easily. With a download manager users can download station-to-archive files from the BSRN ftp-server to the local computer system.

Find the recent version and reference of the BSRN Toolbox at 

BSRN Toolbox is licensed under the GNU General Public License (GPL) and is freely distributed. (Source-code of the BSRN Toolbox as zip-archive, 440 kB)

Reference and detailed description


Downloading station-to-archive files After starting the BSRN Toolbox choose Station-to-archive->Download Station-to-archive files.... The download manager will be opened (Fig. 1). The user can select the station(s), the month(s), and the year(s). The files from this selection will be downloaded to the directory selected under Download directory. With the options Decompress and Check after download the user can specify that the files will be decompressed with gunzip and checked with the BSRN f_check program automatically after download (gunzip and f_check are located in the BSRN Toolbox program directory). If Check availability only was chosen, the BSRN Toolbox will create one file per selected station. Each file contains a listing of the ftp server directory. Before the download can be started the account must be set. Ask [mailto:Gert.Koenig-Langlo@awi.de Gert König-Langlo] for the BSRN ftp-account.

Metadata Select one or more uncompressed station-to-archive file(s) with File->Select File... or File->Select Folder... or by dragging the file(s) onto the program window. Then choose one or all logical records included in the station-to-archive file(s) with the menu Metadata. The BSRN Toolbox will extract the selected metadata. For each station-to-archive file and logical record a file will be created. Behind the metadata information the PANGAEA ID follows. This ID is used for the metadata inside the relational data modell of Pangaea. If this ID is equal to -999, no entry was found and the user has to edit the file named BSRN_IDs.txt on the PANGAEA web server. Only [mailto:Gert.Koenig-Langlo@awi.de Gert König-Langlo] and [mailto:rsieger@pangaea.de Rainer Sieger] have permission to change this file. All other users - who do not import data into PANGAEA - can ignore the PANGAEA ID.

Data Select one or more uncompressed station-to-archive file(s) and choose one or all logical records with the menu Data. For each station-to-archive file and logical record a file will be created. All time series files can be opened with PanPlot. The radiosonde profiles file can be visualized with Ocean Data View (ODV).

Import For the creation of PANGAEA import files the menu Import has to be used. Like the menu Data the BSRN Toolbox will create one file for each station-to-archive file and logical record. These files can be imported to PANGAEA by a curator with 4D. If Overwrite dataset was checked, the import files contain the ID of the dataset to overwrite. This does only work if all BSRN datasets are known in the file named BSRN_Dataset_IDs.txt on the PANGAEA web server. Only [mailto:Gert.Koenig-Langlo@awi.de Gert König-Langlo] and [mailto:rsieger@pangaea.de Rainer Sieger] have permission to change this file.



Tools The BSRN Toolbox provides some useful tools for managing plain text files.

Concatenate files If the user has selected more than one file, this tool concatenates all files. The new file is named ConcatenatedFiles.txt. Starting with the 2nd file a number of lines will be skipped at the beginning of each file. See fig. 2.

Convert Windows End-of-Line to UNIX Station-to-archive files must be in UNIX format. This tool converts the Windows End-of-Line characters to the UNIX End-of-Line character.

Convert MacOS 9 End-of-Line to UNIX Station-to-archive files must be in UNIX format. This tool converts the MacOS 9 End-of-Line character to the UNIX End-of-Line character.

Compress files with gzip</H4> Station-to-archive files must be compressed with gzip. This tool compresses all selected files with the program gzip.

Convert tab files to unformatted format</H4> Unformatted files are using a character for splitting fields. With a dialog the user can specify the field delimiter (comma, semicolon, space or tab stop) and the missing value for empty cells. The input file must be tab separated. See fig. 3.

Convert tab files to formatted format</H4> Formatted files are using columns with fixed width for splitting fields. With a dialog the user can specify the field alignment (left or right), the width of fields and the missing value for empty cells. The input file must be tab separated. See fig. 4.

Quality Check</H3>

As the WRMC was hosted at ETH Zurich, a central flagging of the data quality was performed. The flags and the derived global radiation (calculated from direct and diffuse) were included in the station-to-archive files. Since the transition of the WRMC from ETH Zurich to the AWI the central flagging of the data quality is terminated. All the station-to-archive files in the WRMC have now exactly their native format as given from their station scientists who are responsible for the quality of the submitted data.

Alternatively, the data quality can be controlled now individually by using a new version of the BSRN Toolbox which will be released in the beginning of 2012. The quality checks offered by the BSRN Toolbox can be used by
 * the station scientists to test station-to-archive files before they get submitted,
 * the data curator from the WRMC before archiving submitted data, and
 * by any user of the WRMC testing extracted data from the WRMC.

Input data</H4> Qualitiy checks can be performed with a variety of input data: Quality flags</H4> The implemented qualitiy flags follow the BSRN Global Network recommended QC tests, V2.0 from C. N. Long and E. G. Dutton. They are orderd with respect to the following hierarchy:
 * Any file(s) downloaded from PANGAEA.
 * Many products from the Data Warehouse. This is recommended if the quality of a long time series of a single station will be checked. Of course the parameters which are needed in the check must be selected and no average should be performed.
 * Station-to-archive-file(s). They mast be converted using the "Basic and other measurements, LR 0100 + LR 0300" option from the Data-menue. The station-to-archive-file(s) may be
 * downloaded from the ftp-achive using the Downloading station-to-archive files-menue or
 * produced locally e.g. by a station scientists.

For any measurement only the highest flag is set. Measurements which do not fall into the pyhsically possible limits (flag 5,6) should be taken as highly suspect. Thus, this test should be performed in any case before submitting, archiving or using these data. Measurements beyond the extremely rare cases should at least be visually inspected. If no physical reasons - such as multireflections, extreme weather conditions, etc. - can be found it is recommended to exclude these data from submitting, archiving or using. By comparing measurements with each other (flag 1,2) is is unclear which of the two values cause the inconsistency. Further investigations are recommended. Example</H4>
 * Flag 6: Measurement exceeds the physically possible limit.
 * Flag 5: Measurement fall below the physically possible limit.
 * Flag 4: Measurement exceeds the extremely rare limit.
 * Flag 3: Measurement fall below the extremely rare limit
 * Flag 2: Compared with a corresponding measurement the measurement is to high.
 * Flag 1: Compared with a corresponding measurement the measurement is to low.
 * Flag 0: All tests passed.
 * Download dataset http://doi.pangaea.de/10.1594/PANGAEA.759486 on your local computer.
 * Select the resulting file "GVN_radiation_2011-01.tab" into the BSRN Toolbox.
 * Choose the Quality Check Options from Fig. 6.
 * As a result you will get the file GVN_radiation_2011-01_QC_flags.tab showing that all measurements carry flag 0 which means that all data fall within the physical possible limits. In any other case data users are strongly recommended to exclude these data from further analysis! This can be done by selecting the output option "Cleaned data", see Fig. 6. Station managers should prepare a corrected station-to-archive file and check it again before submission.
 * Repeating the test by selecting "Extremely rare limits" will also result in file GVN_radiation_2011-01_QC_flags.tab now containing some flags 3 and 4. The file can easily be visualized using Panplot, see Fig. 7.
 * It is up to the user (station manager) how to proceed. At least a visual inspection is needed to decide whether the data should be excluded or kept.
 * Quality check option "Comparisons", see Fig. 6 should be used in the same way as the option "Extremely rare limits"
 * All three check options can be selected at the same time. For a more detailed check single checks lead to clearer results since the output file contains just the highest flag of any timestamp.
 * Two algorithm for astronomical auxiliary data are offered. Iqbal 1983 do not regard refraction effects which can be significant especially close to the horizon, Reda 2008 need additional input data such as surface pressure, etc.

Contact: Dr. Rainer Sieger, Alfred Wegener Institute, Bremerhaven