Intern:WRMC Curator

The following information are specially made for data curators who import new measurements and perform the WRMC.

Contact: Dr. Amelie Driemel, Alfred Wegener Institute, Bremerhaven

General Workflow
(1) Station scientists submit station-to-archive file(s) to the internal ftp server ftp.bsrn.awi.de

(2) Automatic mail is send to WRMC Curator that new file(s) arrived

(3) WRMC Curator moves files to the public ftp server and downloads the file(s)

(4) WRMC Curator checks the file(s)

- if quality ok: PANGAEA Import file is created

- if quality issues or format errors are found: station scientist is contacted and asked to resubmit (current pub file is moved to a folder "OldVersionyyyymmdd")

(5) Data available on the ftp server and in PANGAEA

Data conversion and import to PANGAEA
(1) Move new incoming files from ftp.bsrn.awi.de/incoming to pub (2) Download, unpack, and check new station-to-archive files with the BSRN Toolbox (3) Metadata -> create all metadata files (4) Data -> create all data files
 * Windows cmd: putty account@ftp.bsrn.awi.de OR c:\\tools\putty.exe ftp.bsrn.awi.de; macOS: ssh account@ftp.bsrn.awi.de (ask Amelie Driemel for account)
 * cd ../incoming
 * lsbsrn (= provides a list of all folders and shows where new files have been deposited)
 * cd ../xxx/incoming (xxx = station ID)
 * smv * ../../../pub/xxx (moves all files from the xxx/incoming folder on the internal server to the pub/xxx folder on the public server)
 * exit
 * Station-to-archive -> download Station-to-archive files
 * if a PANGAEA metadata ID is missing (station, staff or method) ->
 * create missing entries in accordant 4D tables and
 * edit the BSRN-ID file (on store.pangaea.de/config/BSRN)
 * start the BSRN Toolbox and again create all metadata files

(5) Check data files: for LR0100 use the quality check of the BSRN Toolbox, see https://wiki.pangaea.de/wiki/BSRN_Toolbox#Quality_Check; the resulting file and the datafiles of the other logical records can be plotted with PanPlot (LR0100, LR0300, LR0500 etc.) or ODV (LR1100, radiosonde)

(6) Import -> create all import files

(7) Import files with 4D

(8) Check appearance and validity on the Internet and inform PI

(8) Update php tables by editing files like BSRN_LR0100.txt and BSRN_Status.txt (on store.pangaea.de/config/BSRN)

Setting up a new BSRN-station

 * (1) To participate within the BSRN a station has to fulfill certain conditions.
 * (2) To become a BSRN-candidate the station must formally be accepted by the [mailto:Chuck.Long@noaa.gov BSRN project manager]. A brief overview document of the basic requirements of a BSRN station can be found here.
 * (3) The station manager from the BSRN-candidate station should provide [mailto:mailto:Amelie.Driemel@awi.de Amelie Driemel] with basic station/staff information.
 * (4) In return [mailto:Amelie.Driemel@awi.de Amelie Driemel] sends ...
 * the station manager the individual ftp-account for data submission from the new station,
 * the station identification number and further information needed to produce the station-to-archive file.
 * (5) Within the WRMC the data curator ...
 * imports the new station into PANGAEA as "Event" which automatically updates the BSRN stations table,
 * imports the new station/staff/etc. information into PANGAEA,
 * updates the BSRN-ID-file
 * updates Stations.kmz,
 * asks [mailto:Wolfgang.Cohrs@awi.de Wolfgang Cohrs] to update the maps,


 * (6) The station manager creates, checks, compresses, and submits the first station-to-archive-file to ftp.bsrn.awi.de.
 * (7) After importing the first station-to-archive file into PANGAEA a new line for the new station must be included at the right alphabetic order in all files like BSRN_LR0100.txt and BSRN_status.txt.

Update Stations.kmz
Minor changes can be done manually according to the following instructions: Alternatively, Pan2Applic may be used. As inputfile the output of https://dataportals.pangaea.de/bsrn/stations is a good starting point.
 * (1) download Stations.kmz on your local computer,
 * decompress the local version of the file (*.kmz-files are in fact *.zip-files). The result is a file named BSRN_Stations.kml,
 * edit BSRN_Stations.kml,
 * compress BSRN_Stations.kml into Stations.zip (e.g. using 7-Zip),
 * rename Stations.zip into Stations.kmz,
 * replace Stations.kmz in the web.

If the upper mentioned workflows do not succeed try opening the Stations.kmz in Google Earth, apply the edits and then go to "File - Save place as " (be sure to have marked Stations.kmz on the left side).

Updating PANGAEA data/New Versions of data
30 days after the import a permanent doi-number is attached to a dataset. Before, a dataset can be deleted without problems. Later, a dataset cannot be deleted anymore. If a new version of the dataset - carrying a new doi-number - exists, the old version has to be "hidden" from users by doing the following operations in 4D: Datasets, which have a replaced by reference link to another dataset are removed from the search results and can also not be downloaded via the data warehouse. If someone already cited the old version, however, the citation link still leads to the old version. Therefore, the title of old version has to be changed to "..., Version 1" and in the Commment field of the new version the information on what was changed and when has to be inserted (e.g. "Version 2, 2017-01-10, faulty LWD values corrected"). See also here for an example.
 * searching for the old dataset to be hidden (e.g. search for the *ID in the "Data Set" window) and open it
 * in the 'References' tab click on Add/Remove in the lower part (References for PANGAEA data sets)
 * search for the new version and insert it here by double-clicking on it and then pressing OK
 * choose "Replaced by" in the Relation column and press on save