Intern:MOSAiC raw data

Page content (please note)


This page and the information provided here is addressed to PANGAEA curators and PANGAEA employees only. For PANGAEA user and data submitters please refer to https://wiki.pangaea.de/wiki/Bathymetry (external use only).

(Note, this page is currently under construction and will be updated continuously)...if you have questions, please contact Daniel Damaske.

DAM Underway research data project background


(more information is provided as soon as possible.....link to DAM wiki page)

Standardized content of data sets
To make data findable (F) and due to PANGAEA achieving SOPS'a bathymetry datasets will be separated into two data sets, if the water column information (e.g. wcd files from Kongsbergs MBES Systems) is simultaneously being recorded during bathymetry data acquisition. This is necessary due to data curation processes, consequently reducing the download volume if it is intended to download only bathymetry data (containing only the seafloor reflector information) from PANGAEA. The water column data are usually about 5x larger than regular bathymetry files. (However, datasets will be connected / linked to each other)

Standardized Authors
see https://wiki.pangaea.de/wiki/Bathymetry#Standardized_Authors

(more information is provided as soon as possible.....)

Standardized Title
see also https://spaces.awi.de/pages/viewpage.action?pageId=330337725

To ensure the Reusability (R) and Interoperability (I) of data sets, metadata of similar data sets such as bathymetry data mostly gathered during specific research missions from research vessels or other researh platforms, should follow a common template and using common vocabulary. This is also including the title.

Therefore, the 'title of bathymetry data shall follow a certain template “construction kit”''. See Bathymetry for explanation of this “construction kit” and examples of titles.'''

Standardized Abstract
The abstract should include all necessary information, as described here: https://wiki.pangaea.de/wiki/Abstract

Instructions for the user can be found here: Bathymetry

Raw data (example)
''Swath sonar bathymetry data using Kongsberg EM 122 multibeam echosounder was recorded during RV METEOR cruise M127 (WHAT, HOW). The cruise took place between 25.05.2016 and 28.06.2016 in the North Atlanic Ocean (WHEN). The survey data contain MBES measurements of the Fictional Bank canyon. No bathymetric measurements of Fictional Bank canyon were found in international databases prior to the cruise (WHY).

''To enhance MBES data accuracy, two sound velocity profile casts were conducted in the vicinity of the working area prior to the survey using sound velocity profiler AML Oceanographic (WHAT, WHEN & WHY). After processing, these data were directly imported into the MBES acquisition software Kongsberg SIS Seafloor Information System.

(more information is provided as soon as possible.....)

Processed data (example)
''Swath sonar bathymetry data used for that dataset was recorded during RV METEOR cruise M127 using Kongsberg EM 122 multibeam echosounder (WHAT, HOW). The cruise took place between 25.05.2016 and 28.06.2016 in the North Atlanic Ocean (WHEN).

To enhance MBES data accuracy, two sound velocity profile casts were conducted in the vicinity of the working area prior to the survey using sound velocity profiler AML Oceanographic (WHAT, WHEN & WHY). After processing, these data were directly imported into the MBES Acquisition software Kongsberg SIS Seafloor Information System (WHEN & HOW). Raw data were being converted to MB-System format. Data were manually edited for false measurements No further sound velocity profile were used during processing of this dataset. A raster was calculated and stored in GeoTIFF format with a 25m resolution (negative values).

(more information is provided as soon as possible; feel free to provide more detailed processing information here, but keep it as compact as possible.....)

Standardized usage of Reference
For data from german research vessels, the cruise report should be listed under References as Related to.

Datasets of processed data and raw data from one entire cruise should be linked as Related to to each other. Currently, all bathymetry datasets already archived in PANGAEA are reviewed and checked.

Standardized Coverage
Automatical entry in PANGAEA calulated from Events or PANGAEA Geocode (see below)

https://wiki.pangaea.de/wiki/Metaheader

Please note, that Minium Elevation und Maxium Elevation does not represent min / max depth values from MBES data contained in this dataset.

Note: Currently, the Coverage will be visable on PANGAEA website with points having an GoogleMaps image in the background (right upper corner). It does not show the actual coverage of the surveyed area. The points are representing the Events.

Underway event
Each German campaign (since ~2016) should include so called "Underway" Events that are created on board. The "Underway" Event is named like the following example: CruiseID_0_Underway-sequential number e.g. M127_0_Underway-1 (this schema was established during the MANIDA project). If this Event was not created on board of the research vessel, please check in the station list which is the next free number to create an "Underway" Event. Please do not use the track Event anymore!

This Event has to include the Device used to record the data and latitude/longitude and date/time for start and end of file recording needs to be updated in order to show the first and last data record of the MBES device acquired during the cruise. This information can be derived from the parameters table that is uploaded together with the data file links (see Intern:Bathymetry).

If available, an additional link to the Sensor description hosted on https://sensor.awi.de should be included.

SVP event(s)
If SVP casts (sound velocity profile measurements using an external device e.g. using an AML Oceanographic Plus X sound velocity probe) were made during the cruise, the data files which were acquired (and used for import to the acquisition software, e.g. the Kongsberg Seafloor Information, SIS) during the cruise will be stored within the same raw data set. The corresponding event of the Event list of the cruise (e.g. https://www.pangaea.de/expeditions/events/M160) will be linked to these files. This Event will contain similar information as discribed above, e.g. coordinates,methods/devices, etc.. If SVP's are used for data post-proesssing (sound veleocity correction), these data files can be also submitted to PANGAEA. If possible, these SVP can be also linked to the corresponding Event.

Standardized usage of Parameter
Please note, that a script must be executed on AWI-Server to get the parameters (the script is currently under development). Please contact Daniel Damaske.

Raw data curation
Standard parameters for bathymetry raw data include also PANGAEA’s so called Geocodes (https://wiki.pangaea.de/wiki/Geocode). These are standard important parameters, which are mandatory to keep the data georeferenced. These metadata for each file will be automatically created while data curation process.

(more information (and more additional parameters) is provided as soon as possible.....).

Processed data curation
Standard important parameters, which are mandatory to keep the data georeferenced and described will be automatically attached while data curation process.

(more information is provided as soon as possible.....)

Standardized usage of Comment
Always include "These data should not be used for navigational purposes" as dataset comment for each bathymetry dataset.

For further information see Bathymetry.

Licence
PANGAEA and DAM are encouraging data submitters that bathymetry data are submitted with CC-BY 4.0 License terms, which allows share (copy and redistribute the material in any medium or format) and adapt (remix, transform, and build upon the material) the data (https://creativecommons.org/licenses/by-nc/4.0/) This allows also PANGAEA / DAM in future to make data coverage and the trackline visible at https://marine-data.de/ with a direct link to PANGAEA. Within the DAM efforts, coverage and trakline shall also be made available to the community via OGC Web Map Services (WMS). Avoid licence like CC-BY-ND-NC if possible.

Which CC license choose for data? Check also https://wiki.pangaea.de/wiki/License



Keywords/Project
If the data is direct DAM data (from a DAM member, cruise or from BSH) add "DAM" as project label. If the data is not directly linked to the project, but this standard workflow was used, please add the "DAMUnderwayResearchData" as technical keyword.

Export table file name
The filename for the export table (not the data itself) should follow a certain structure and all letters should be in lower case. This structure is: Campaign_mbes_datatype_device_datasettype. Example: msm88_1_mbes_raw_em122_ed.


 * datatype: raw or processed
 * device: em122 or em712 or...
 * datasettype: ed (entire dataset) or wad (working area dataset)

Raw data
https://doi.pangaea.de/10.1594/PANGAEA.918716 (Please note that the Abstract is currently still missing, dataset currently in review, therefore not fully accessible)

Processed data
https://doi.pangaea.de/10.1594/PANGAEA.912259 (Please note, this is a test dataset, dataset currently in review status, therefore not fully accessible)

(Standardized usage of Data model extensions) (for raw and processed data curation)
The usage of so-called data model extensions which are further attributes is currently developed. This is a new feature for PANGAEA in order to include more standardized metadata. This new feature shall insure further findabilty (F) and interoperability(I) of the data sets (more information is provided as soon as possible.....).

Standard operating procedure for uploading bathymetry data
Here is a step-by-step workflow for upload of bathymnetry data.

Following steps are currently done (workflow status Nov. 2020):
 * When a data submission comes in via ticket, provide an upload link
 * When all data is uploaded, sub-task is created that is assigned to Daniel Damaske
 * Create a "dummy" dataset with the standardized title and authors and write the dataset ID into the subtask
 * Check if "Underway" event exists, if not create one, and write the event in the subtask
 * If there is information for a moratorium, write this into the subtask
 * Daniel Damaske will now produce an import file with the standardized parameters. It takes several hours to run the script, if all goes well.
 * As soon as the import file is provided, the dataset can be uploaded/overwritten with the real data. You might need to add event information, file content, latitude, longitude and date/time for svp or other additional files prior to the upload.
 * Link the Cruise report as "further details"
 * Add standardized export table name and standardized comment

_______________________________________________________________________________________________________________________________________________________________________________________________

Old content from external page
Old content from external page temporary stored here (deleted from external page)

Data from bathymetry may be archived in different versions of processing. All data are password protected by default unless free access of the author is officialy defined through the related ticket.

Processing

 * 1) Original raw data as produced by the swath sonar system in its proprietary format. A link to a software or a description for conversion to ASCII or standard format is recommended. doi:10.1594/PANGAEA.767553
 * 2) Different versions of processing steps.
 * 3) Processed and corrected but measured values in ASCII-format (latitude/longitude/depth i.e. XYZ) - use parameter Depth, bathymetric (2268). doi:10.1594/PANGAEA.418153
 * 4) Processed digital elevation model in ASCII-format (latitude/longitude/depth) - use parameter Depth, bathymetric, interpolated (102737). doi:10.1594/PANGAEA.832573
 * 5) Final map in pdf-format; optional also in original grafic (e.g. Illustrator) and/or GIS format. doi:10.1594/PANGAEA.805734, doi:10.1594/PANGAEA.805735

Metadata
A description (Metadata) must always accompany each data set:
 * Author(s)
 * Title
 * Year of publication
 * Institute as source
 * Method: type of sonar system
 * Description of processing steps and status and possible errors
 * Abstract (for final maps or in bathymetric data publications)

Workflow
workflow for bathymetry data archiving
 * on board data are stored in ships tape-drive
 * after cruise data are stored in land tape-drive (hs/datex); overview via http://ingest.awi.de/
 * SCRIPT to validate file structures
 * SCRIPT generates tar-archive
 * SCRIPT generates Pangaea import files
 * SCRIPT copies data to hs/usero/bathy (bathymetry) and hs/usero/para (parasound)
 * PI opens ticket at http://issues.pangaea.de/ with metadata
 * PI uploads import file to ticket
 * import to Pangaea by curator

Device description with formats at https://spaces.awi.de/
 * Bathymetrie - https://spaces.awi.de/confluence/display/PSdevices/Faecherecholot
 * Parasound - https://spaces.awi.de/confluence/display/PSdevices/Sedimentecholot

to be solved

 * Data authors do not care about the storage format
 * Data were not stored in tape-drive
 * Format problems while generating Pangaea import files

PANGAEA HS/bathy Folder Structure
The following folder structure shall be used for /bathy (only lower case letters alowed, only underscore sysmbol):


 * 1.) cruise/  (e.g. ps81/ or msm88_2/)


 * 2.) mbes/ or water_column/ or side_scan/


 * 3.) raw/ or processed/


 * 4.) em122/ or em712/ or ds3/ or ....


 * 4.) entire_dataset/ or transit_dataset/ or working_area_dataset/ or dataset_compilation/


 * 4.1) if working_area_dataset/ or dataset_compilation/ make subfolder with IHO-IOC GEBCO undersea feature name'' (e.g. Ascension_Fracture_Zone)