Intern:Import

Workflow for the import of data with the Intern:4D client.

For Workflow new Editorial System see also here. This presentations describes single dataset upload and list imports. Here you can find mass import of datasets and references.

single data set

 * First column: Event label as header, followed below by the label as defined in the 4D event table.
 * Next columns may contain the third spatial dimension (e.g. DEPTH, sediment) or the time (DATE/TIME) with ID/parameter name in header line; for options see geocode.
 * The following data columns should contain in the header line
 * the parameters full name as defined in the parameter table followed by a blank and the unit in bracket [] or
 * the parameters short name followed by a blank and the unit in bracket [] or
 * the parameter ID

Ambiguous parameters can be made unique with the unit in brackets attached:



If you want to already add a parameter comment, a method ID or the column format to the import sheet you can use the following "add-ons" after the parameter name and unit:


 * //*comment
 * //$methodID
 * //%format
 * //!PI/StaffsID
 * //#terminologyID (for "Term" typed Parameters this allows to specify the terminology, e.g. 1=WORMS, 2=ITIS, from which the plain text terms are looked up)

You can also use two or all of them, just put them one after the other e.g.:

Calcium [mg/kg]//*1st measurement//$125//%#0.000



Save your table as a tab-delimited textfile (*.txt). In 4D, you can import your textfile clicking on Import > Data. The import window opens (Basics tab), allowing you to select your file by clicking on Open.

After selecting your textfile you add Author(s), Reference(s) and Project(s) by clicking in the respective field. Be sure to also change the Title and to check the publication year. If no reference for certain data sets is given (and not to be expected), use the field Source to define the source of the data. Source is relational to the institution table.

In the tab Config you can choose/change the PI and the methods and add a parameter-specific comment as follows:
 * tick the Edit in list box
 * mark the respective cells in the PI/Method-column and click on Choices
 * a new window appears where you can choose the PI/Method by double-clicking or clicking on Select
 * if you want to add a parameter-specific comment just click into the respective cell in the Comment-column and insert the text
 * you can also change the format of the cells (Format column) e.g. by adding digits (#0.00 instead of #0) or removing them (#0.0 instead of #0.000)



Lastly, in the tab Details you can add an Export filename, a data set Comment or the Abstract. If a single data set is a supplement to an article (Status supplementary data in the Basics tab), an abstract has to be provided. In this case the Export filename should be in the format Author_Year, 1stAuthor-2ndAuthor_Year or Author-etal_Year.



To finalize the import process, there are two possibilities, you can..

... overwrite an existing data set (be careful!)

Please overwrite existing datasets only if the DOI registration was not yet completed or if the changes in the dataset can be and are documented in the dataset history!

If an already existing data set (i.e. already imported into 4D) has to be replaced, you have to change the ID of your data set before clicking on the Import button. Just click on Change besides the Data set ID field on the very top of the Import Window, and then on Replace. A window then opens which allows you to choose the old data set (the one which needs to be replaced by the new version) via doubleclick - please check carefully that it is the right one!. After having changed the ID you have to check all fields again, because they adapt some of the features (Title, Export Filename) of the old dataset. Then click on the Import button (you will get asked if you really want to overwrite an existing data set).

... import a new data set

If you don't want to replace an old data set just click on the Import button without changing the ID (4D always povides a fresh ID as soon as the Import window is opened).

After this whole process, please always check the metadata and the data table you just imported by clicking on Open URL. Note that import of data sets with special characters (öäüß) does not work with Macintosh 4D-clients due to an ODBC driver problem.

several data sets (Massenimport)
For single files an editor or spreadsheet software is used (see above), for multiple files with similar format, the use of PanTool and Split2Events, as well as Jupyter Notebook templates are recommended.

Using 4D for import many similar tables
This can be used for importing multiple tables with the same header.
 * Select the files for the import via the Open button
 * Select the settings (JSON) file for the settings via Menu -> Load settings. Alternatively, fill the first opened import file with the desired meta-info, also set placeholders here where needed.
 * Placeholders %event%, %filename%
 * In case of a dataset without the event column, when a file name is equal to a Event label present in PANGAEA, it is automatically added to the dataset. Note: it is only visible in the dataset after the import!
 * Use Menu -> Save settings into selected file[s] to fill all selected files with the JSON header.
 * Select the finished import files via the Open button and start importing.

Recipies for the import of various data types

 * Vertical sediment profile
 * Geological Age
 * Reflection seismic
 * Echo sounder
 * Bathymetry
 * Mooring
 * Pollen
 * CTD