Intern:Data set

A data set is a collection of data (often from one event) in a scientific context organized in one matrix. Data in Pangaea are organized in predefined data sets which are quite similar to the original files uploaded and exported from the archive.

The granularity of a data set depends on the type of data, the number of data points and is primarily in the decision of the data author. In principle a Pangaea data set can have an unlimited number of columns and lines (excel 2003: 65,536 x 256; excel 2008: >1 Mio x 16,384) - Examples: A data set may contain one to many data series. Two to many data sets may be grouped to one parent set. Access rights can be defined for a complete data set only. Each data set consists of the data accompanied by metadata according to ISO standard fields (ISO 19115). A data set appears on the Internet with a metaheader which contains the information as described below.
 * 507 columns
 * 2,000,000+ lines
 * 22,600,000+ lines (in ascii: 551 MB; in ASE +index: 2.2 GB; export from IQ +DOI: 1.44 GB) (Fig. 2)

Opening a data set in 4D will show the frame title and five tabs named Basic, Config, References, Details and Coverage with metadata fields as described below:

Frame title
The frame title of the data set window shows its ID and the responsible curator (s. Fig. 1). Below are buttons to  and  the data set or to open it via its URL in a browser window . The button  is offered when the data set is part of a parent set and allows to open the corresponding parent set. Checking the with cache new box will force a complete update in the data file cache when pressing .

Basics tab (Fig. 1)

 * Author(s) of the data set; one to many authors may be added by a multiple choice list related to the table Staff
 * Year, automaticaly set but can be changed
 * Title of the data set as free text; equivalent to the title of a publication; in case the data set is a table from a publication, the title should be the table caption, headed by the table numer, e.g. doi:10.1594/PANGAEA.693592
 * Source may contain the institution of the data origin and is relational to the table Institution; use only if data are not related to a reference.
 * Status of the data set with
 * pop-up menue with choices: questionable, not validated, validated, published, published & citable
 * Registry: gives information about the registration process:
 * not to be registered if status is not published
 * registration is in the lead time for four weeks after setting the data set to status published and final editing
 * registered as the final status
 * Protection of the data set with
 * pop up menue with choices: unrestricted, signup required, access rights needed
 * unrestricted open access data sets (default)
 * signup required for e.g., BSRN datasets
 * access rights needed data sets under moratorium. Login required may be checked for sets with status published but should still be protected (on request of the PI)
 * Access rights button to set individual access to data sets with access rights needed
 * License: pop-up menue with different cc creative commons license, CC-By by default, see Creative Commons Attribution-Noncommercial. http://i.creativecommons.org/l/by-nc/3.0/88x31.png
 * Keywords is relational to the Thesaurus and can be set individually, different types of keywords are available
 * Project(s) allows via a mutiple choice list to add one to many projects as provided by the Project table
 * Event(s) as used in the data set (can not be changed)
 * Created date/time of import; Updated date/time of last change

Config tab (Fig. 3)

 * Data series window shows parameters used in the data set. The list contains data series label, parameter name, unit and original format. The button Add/Remove allows to add or remove data series to the data set. Data series must be deleted from the configuration before they can be removed.
 * Geocodes window lists the used geocodes. A double-click brings a geocode in the data set Configuration.
 * Related metainformation window contains fields from the event table which can be added to the Configuration.
 * Configuration window lists geocodes (highlighted in blue), related metainformation (highlighted in yellow) and all parameters used in the order of the available data set. The configuration lists shows the data set label, parameter name, unit, format, parameter ID, PI, Method, Comment and no. items. Mark the Edit in list check-box to edit these items.
 * Format will show the number of digits before and after the decimal point of a numeric parameter if selected in the configuration window by a mouse click. Different formats can be selected from the pop-up menue or changed by hand. If the geocode Date/Time is selected, different types of ISO formats can be selected, depending on the required precision. Also the exponential-format is supported, e.g.
 * PI and Method are relational fields and can be set via the button choices from the relational lists
 * Parameter can be changed in the same way via the choices button. Be carefull, never exchange a text parameter by a numerical parameter or vice versa! Unit and Param.ID are linked with the parameter and cannot be changed.
 * Comment is a free text field

References tab (Fig. 4)

 * References (literature & data) is related to the reference list and offers all kind of references from literature, web pages and external data repositories.
 * Add/Remove opens the relational reference list.
 * Choices is inoperable
 * The reference list contains Author(s), Title, Year and ID of the related reference. Comment is a free text field. Relational opens a pop-up menue with the options: Replaced by, Supplement to, Related to, Other version, New version, Original version, Source data set and Further details.
 * Replaced by
 * Supplemet to
 * Related to by default
 * Other version
 * New version
 * Original version
 * Source data set
 * Further details
 * References (Pangaea data sets) is related to the Pangaea data sets list. Data sets can be referenced in the same way as the references (literature & data).
 * Add/Remove opens the data set list.
 * Choices is inoperable
 * Relational opens the same pop-up menue as describe in references (literature & data)

Details tab (Fig. 5)

 * URL as defined by the system for event-related data sets or as defined by the user for static links to files. For static links the check-box static must be selected.
 * Export filename contains the data set's name if downloaded as text file to the users PC. The extension *.tab is added automaticaly.
 * Filenames usualy start with the event, followed by a specification of the content (e.g. M24_3-5_sedimentology).
 * File names of supplements start with the first author's name followed by the year, equivalent to the citation of references in a publication text; e.g. Smith_1998, Smith-Sandwell_1987 or Smith-etal_2007.
 * Comment to add individual comments as plain text; field size up to 32 kbyte. URIs might be included and will be resolvable in the metaheader (example ).
 * Abstract to add the (paper's) abstract. Is mandatory for supplementary to data sets.

Coverage tab (Fig. 6)

 * Spatial coverage: fields showing min/max of the three spatial dimensions of the data set. Set automatically; data sets without the three dimensions get by default Latitude min./max. -90/90, Longitude min./max. -180/180 and 3. dimension min./max. -2000/20000. These values can be deleted.
 * Size of the data set, set automatically and gives Rows number, Columns number and Data points.
 * Temporal coverage: min/max of Date/Time or Age [ka]
 * Topologic type is used to define the extension of a data set

Locked data set or other entries
When opening a window to edit details of a record, the record is locked and is not available for other users and also not for background processes. As any updated records are processed sequentially in the background queue, this might cause updates not showing up on the web. Close windows if editing is done!

Deleting/Updating PANGAEA data
30 days after the import a permanent doi-number is attached to a dataset (only published and published & citable status). Before, a dataset can be deleted without problems. Later, a dataset can be deleted only, if a new version of the dataset - carrying a new doi-number - exists.
 * 1) Import the new version of the data set
 * 2) Open old version, go to References tab.
 * 3) Choose the new data set in References (PANGAEA data sets) and set the Relation replaced by.
 * 4) save the old version
 * 5) reopen the old version and delete it