Export

From PangaWiki
Jump to: navigation, search

Data can be retrieved and exported from Pangaea in various ways.

  • PANGAEA search (on the home page at http://www.pangaea.de)
  • DDI (Direct Downlaod Interface)
  • DOI (Digital Object Identifier)
  • ART (Advanced Retrieval Tool)
  • through Portals, established by using panFMP
  • Data warehouse (operational since 2008-07-18, admin only)

Mass download for compilation and visualization

ODV map of amphibole distribution compiled from 900 data sets in Pangaea

Example: How to create a map of heavy mineral distribution of the Arctic Ocean

0. Install PanGet (http://www.pangaea.de/Software/PanGet), PanTool (http://www.pangaea.de/Software/PanTool) and Pan2Applic (http://www.pangaea.de/Software/Pan2Applic) first.

1. Retrieve datasets with Pangaea Search Engine (http://www.pangaea.de). For details read the PangaVista Help (http://wiki.pangaea.de/wiki/PangaVista)! In this example we create two querys:

  • 1.1 hornblende or amphibole or epidote or garnet or pyroxene or clinopyroxene or orthopyroxene or opaque and latitude > 60
  • 1.2 Quartz or Feldspar or kalifeldspar or plagioclase or calcitev or dolomite and latitude > 60

2. Create a download script with PanGet. For details read the PanGet manual (http://wiki.pangaea.de/wiki/PanGet)!

3. Start the download script. For details read the PanGet manual.

4. After downloading all datasets from PANGAEA verify the files with PanTool.

  • 4.1 Select the data folder (Ctrl+O) and choose "Verify PANGAEA dataset ..." from the "Special tools" menu (F11). PanTool writes the file "VerifyResults.txt".
  • 4.2 Open this file with an editor (e.g. Excel) and report lines marked as error to info@pangaea.de. We will fix the errors as soon as possible.
  • 4.3 When the PANGAEA team has fixed the errors start step 3 again.

5. Excursus

  • 5.1 Open the option dialog of the program PanTool (Ctrl+P) and set the filename format to "zz%a_%N.txt".
  • 5.2 Open five random datasets with an editor like Excel and try to understand the general structure of PANGAEA datasets.

6. Extract parameter lines from all data files

  • 6.1 Select the data folder with PanTool (Ctrl+O).
  • 6.2 Choose "Search one string ..." from the Menu "Basic tools" (F5). Search for "PI:". You get a file named "ResultOfSearch.txt".

7. Make a list of all parameters, methods and comments

  • 7.1 Select the file "ResultOfSearch.txt" with PanTool (Ctrl+E) and choose the tool "Search and replace one string ..." (F8). Search for " * " and replace this with "^t" (^t will be replaced by the tabulator).
  • 7.2 Open "zz1_ResultOfSearch.txt" with Excel and produce a list with three columns (Parameter, Method, and Comment). Sort by parameter, method, comment. Add a column after "Comment" and give "x" as value (This will fix an Excel optimize problem).
  • 7.3 Select "zz1_ResultOfSearch.txt" with PanTool and choose the tool "Delete double lines". The new file (zz1_zz1_ResultOfSearch.txt) contains all parameters and methods once.

Open it with Excel and see the problems: alike parameters have different names (e.g. "Hornblende" = "Amphibole"), alike methods are described with different precision (e.g. "X-ray diffraction, silt fraction" vs. "X-ray diffraction"), comments can be expressed differently too (e.g. "<63 micron" vs. "less than 63 µm"). So, datasets should be harmozied before you can create a map!

8. Harmonize parameter names, methods and comments in all datasets with search and replace

  • 8.1 Find the problems in "zz1_zz1_ResultOfSearch.txt" and create a "search and replace database". First column: string to search for, second column: string to replace it with. Name this file "HeavyMinerals.rdb". See the example file:
Column 1                                                Column 2
Hornblende                                              Amphibole
Hbl                                                     Amph
X-ray diffraction TEXTUR, clay fraction                 X-ray diffraction
X-ray diffraction, bulk mineralogy                      X-ray diffraction
X-ray diffraction, silt fraction                        X-ray diffraction
Counting, binocular                                     Heavy mineral analysis
Element analyser CHN, LECO (URL: http://www.leco.com/)  Element analyser CHN

9. Apply the PanTool tool "Search and replace many string at once ..." (Shift+F8) to the data folder.

10. Create a data file Start Pan2Applic and set options (Ctrl+T). On card "All" set: "set GEOCODE range", "display DOI", "show method", "show comment". On card "Format" choose "ASCII/tab-separated". Create a new folder and drag all "zz1_*" to this folder. Drag the new folder onto the Pan2Applic window and follow the instructions.