BSRN Toolbox
The Baseline Surface Radiation Project (BSRN) makes use of the so called station-to-archive format to share the data of one station for one month with the BSRN community. This format cannot be understood without reading the BSRN format manual (or see here for a more recent summary). So the BSRN Toolbox was developed to convert this format to more readable dataset tables and for creating PANGAEA import files. These tables can be opened with a spreadsheet program like Microsoft Excel or OpenOffice and with PanPlot to visualize the data very easily. With a download manager users can download station-to-archive files from the BSRN ftp-server to the local computer system.
Find the current version and reference of the BSRN Toolbox at doi:10.1594/PANGAEA.901332
Contact: Amelie Driemel. The software is provided as freeware under the (GNU General Public License (GPLv3) and is freely distributed without warranty by the Alfred Wegener Institute, Helmholtz Center for Polar and Marine Research, Bremerhaven.
Installation
Open doi:10.1594/PANGAEA.901332 and click on "View dataset as HTML". Download the current version of the BSRN Toolbox to your computer.
Windows
Double-click BSRN_Toolbox_Win.exe and follow the instructions. If the MSVCR120.dll is missing on your computer, you have to install the Visual C++ Redistributable Packages for Visual Studio 2013 from Microsoft.
OS X
Open the downloaded dmg file with a double-click. Drag and drop the file BSRN_Toolbox.app onto the appliction folder icon.
Linux - SuSE
Uncompress the archive to your user bin directory. Double-click BSRN_Toolbox.sh.
Linux - Debian 64bit
- dpkg --add-architecture i386
- apt-get update
- apt-get install libgl1-mesa-glx:i386 libqt5gui5:i386
- Uncompress the archive to your user bin directory. Double-click BSRN_Toolbox.sh.
Reference and detailed description
Downloading station-to-archive files
Before a download of BSRN station-to-archive files can be started the account must be set. Ask Amelie Driemel for the BSRN ftp-account.
After starting the BSRN Toolbox choose Station-to-archive->Download Station-to-archive files.... The download manager will be opened (Fig. 1). Here, on the second tab ('Account'), the server address (ftp.bsrn.awi.de) and the received account have to be inserted. In the tab 'Options' the user can then select the station(s), the month(s), and the year(s). The files from this selection will be downloaded to the directory selected under Download directory. With the options Decompress and Check after download the user can specify that the files will be decompressed with gunzip and checked with the BSRN f_check program automatically after download (gunzip and f_check are located in the BSRN Toolbox program directory). If Check availability only was chosen, the BSRN Toolbox will create one file per selected station. Each file contains a listing of the ftp server directory.
Metadata
Select one or more uncompressed station-to-archive file(s) with File->Select File... or File->Select Folder... or by dragging the file(s) onto the program window. Then choose one or all logical records included in the station-to-archive file(s) with the menu Metadata. The BSRN Toolbox will extract the selected metadata. For each station-to-archive file and logical record a file will be created. Behind the metadata information the PANGAEA ID follows. This ID is used for the metadata inside the relational data model of PANGAEA. If this ID is equal to -999, no entry was found and the user has to edit the file named BSRN_IDs.txt on the PANGAEA web server. Only Amelie Driemel has permission to change this file. All other users - who do not import data into PANGAEA - can ignore the PANGAEA ID.
Data
Select one or more uncompressed station-to-archive file(s) and choose one or all logical records with the menu Data. For each station-to-archive file and logical record a file will be created. All time series files can be opened with PanPlot. The radiosonde profiles file can be visualized with Ocean Data View (ODV).
Import
For the creation of PANGAEA import files the menu Import has to be used. Like the menu Data the BSRN Toolbox will create one file for each station-to-archive file and logical record. These files can be imported to PANGAEA by a curator using 4D. If Overwrite dataset was checked, the import files contain the ID of the dataset to overwrite. This does only work if all BSRN datasets are known in the file named BSRN_Dataset_IDs.txt on the PANGAEA web server. Only Amelie Driemel has permission to change this file.
Tools
The BSRN Toolbox provides some useful tools for managing plain text files.
Concatenate files
If the user has selected more than one file, this tool concatenates all files. The new file is named ConcatenatedFiles.txt. Starting with the 2nd file a number of lines will be skipped at the beginning of each file. See fig. 2.
Convert Windows End-of-Line to UNIX
Station-to-archive files must be in UNIX format. This tool converts the Windows End-of-Line characters to the UNIX End-of-Line character.
Convert MacOS 9 End-of-Line to UNIX
Station-to-archive files must be in UNIX format. This tool converts the MacOS 9 End-of-Line character to the UNIX End-of-Line character.
Compress files with gzip
Station-to-archive files must be compressed with gzip. This tool compresses all selected files with the program gzip.
Convert tab files to unformatted format
Unformatted files are using a character for splitting fields. With a dialog the user can specify the field delimiter (comma, semicolon, space or tab stop) and the missing value for empty cells. The input file must be tab separated. See fig. 3.
Convert tab files to formatted format
Formatted files are using columns with fixed width for splitting fields. With a dialog the user can specify the field alignment (left or right), the width of fields and the missing value for empty cells. The input file must be tab separated. See fig. 4.
Quality Check
The BSRN Toolbox features means to perform quality checks of BSRN data. These features are available in the menu bar under Quality Check. For now, there is only one option BSRN Recommended V2.0 available. An option with user configurable limits is planned.
The implemented qualitiy checks follow the BSRN Global Network recommended QC tests, V2.0 from C. N. Long and E. G. Dutton.
Potential users of the quality check feature of the BSRN Toolbox are:
- the station scientists to test station-to-archive files before submission
- the data curators from the WRMC before archiving submitted data
- any user of the WRMC testing extracted data from the WRMC
Input data
Qualitiy checks can be performed with a variety of input data:
- PANGAEA file(s), e.g. as downloaded from https://dataportals.pangaea.de/bsrn/?q=LR0100
- Many products from the Data Warehouse. Be aware that the check limits are intended to be used on high frequency data (temporal resolution of 1-3 minutes). Therefore no averaged data supplied by the Data Warehouse should be used.
- Station-to-archive-file(s) must be converted using the Data -> Basic and other measurements, LR 0100 + LR 0300 option from the Data-menue first. The files created in this manner can then be used as input data for quality checking. The station-to-archive-file(s) may be
- downloaded from the ftp-achive using the Downloading station-to-archive files-menue or
- produced locally e.g. by a station scientist.
Quality codes
The quality codes that are written to the output files are unsigned integer values in decimal notation. Each quality code holds 6 quality flags (each represented by one bit of the quality code). The individual bits mean (from LSB to MSB):
| Bit position | Decimal representation | Meaning |
|---|---|---|
| 0 | 1 | Measurement falls below the physically possible limit |
| 1 | 2 | Measurement exceeds the physically possible limit |
| 2 | 4 | Measurement falls below the extremely rare limit |
| 3 | 8 | Measurement exceeds the extremely rare limit |
| 4 | 16 | Compared with a corresponding measurement the measurement is too low |
| 5 | 32 | Compared with a corresponding measurement the measurement is too high |
This results in quality codes between 0 and 63. A quality code of 0 means that all checks were passed successfully.
Measurements which do not fall into the physically possible limits should be taken as highly suspect. Thus, this test should be performed in any case before submitting, archiving or using these data. Measurements beyond the extremely rare cases should at least be visually inspected. If no physical reasons - such as multireflections, extreme weather conditions, etc. - can be found it is recommended to exclude these data from submitting, archiving or using. When comparing measurements with each other it is unclear which of the two (or more) values caused the inconsistency. Further investigations are recommended.
Example
- Download dataset http://doi.pangaea.de/10.1594/PANGAEA.759486 onto your local computer.
- Select the resulting file "GVN_radiation_2011-01.tab" into the BSRN Toolbox.
- Choose the Quality Check Options as shown in Fig. 6.
- As a result you will get the file GVN_radiation_2011-01_QC_codes.tab showing that all measurements carry flag 0 which means that all data fall within the physical possible limits. In any other case data users are strongly recommended to exclude these data from further analysis! This can be done by selecting the output option "Cleaned data", see Fig. 6. Station managers should prepare a corrected station-to-archive file and check it again before submission.
- Repeating the test by selecting "Extremely rare limits" will also result in file GVN_radiation_2011-01_QC_cleaned.tab now containing some flags (8=Measurement exceeds the extremely rare limit). The file can easily be visualized using PanPlot, see Fig. 7.
- It is up to the user (station manager) how to proceed. At least a visual inspection is needed to decide whether the data should be excluded or kept.
- By selecting "One output file" and choosing "Original data" and "Quality codes", see Fig. 6, a file called GVN_radiation_2011-01_QC_combined.tab will be created. Again, PanPlot can be used to visualize e.g. SWD and the corresponding quality flags SWDQF, see Fig. 8. In this case SWD measurements exceed the extremely rare limit caused by multireflexion between broken clouds and the snow surface. Thus, these data should be kept for further analysis.
- Quality check option "Comparisons", see Fig. 6 should be used in the same way as the option "Extremely rare limits"
- All three check options can be selected at the same time. Single checks are recommended for fast overviews.
- Two algorithm for astronomical auxiliary data are offered. Iqbal 1983 do not regard refraction effects which can be significant especially close to the horizon, Solpos need additional input data such as station pressure and temperature at 2 m height which are not available in all BSRN datasets.
Short-wave downward versus Direct and Diffuse
The difference between the Short-wave downward (SWD) and the sum from Direct (DIR) and Diffuse (DIF) on a horizontal plane is a usable error indicator. The quality check option "Comparisons" offers the necessary output if "Auxiliary data", "Original data", and "One combined output file" are chosen.
Among many other parameters the outputfile *QC_combined.txt contains SWD as well as SumSW which can be displayed e.g. using PanPlot2. More convenient is to use the additional output file *QC_combined_SDW_SumSW.txt which contains just the 2 desired parameters SDW and SumSW which can be displayed with PanPlot2 without any further manipulations.
Contact: Amelie Driemel, Alfred Wegener Institute, Bremerhaven